AfriSyCo measures how question wording impacts African language AI answers
AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
AfriSyCo looks at how changing the way questions are asked affects AI answers in African languages. The researchers found that more confident (assertive) question phrasing leads AI models to pick wrong answers more often. Adding verification steps helps reduce errors but does not fully fix the problem. The exact wording and the AI model version also hugely change how likely the AI is to choose the right answer.
What this means in practice
- •For language technology developers: Improve African language AI question prompts to reduce incorrect confident answers and boost factual accuracy.
- •For chatbot designers: Design chatbot prompts that use verification-related framing to lower mistakes in responses for African languages.
Authors
David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening, Elvis Gyasi Owusu
Abstract
AfriSyCo studies answer switching around African-language factual content with two complementary layers: native-language follow-ups and a controlled cross-language factorial whose question, options, and target remain in the African language while the follow-up framing is English. We analyze 1,415 turn-1-correct model-language-item observations derived from 100 source questions across seven open-weight checkpoints and six languages; turn-1-correct denotes observed first-response accuracy, not demonstrated knowledge. Under native prompts, assertive endorsement produces 29.3 percentage points more any-turn false-target selection than mention-plus-verification (M+V), with a 19.0-point immediate T2 contrast. In the precommitted 2 x 2 factorial, averaged over three tested prompt families, assertive framing increases target selection by 30.4 points (95% CI [28.4, 32.3]); verification decreases it by 17.4 points, while the assertive effect rises from 20.5 points without verification to 40.2 with it (interaction +19.7). The effect remains 34.8 points among 611 observations correct after option reordering. Magnitude varies sharply by wording and checkpoint: prompt-family effects span 20.1-42.5 points, a Twi/Qwen3 paraphrase shifts target selection from 70.8% to 4.2%, and checkpoint effects span 9.2-47.0 points. Prompt realization is therefore part of the measurement problem.