Chain of thought faithfulness increases in low resource languages for knowledge tasks
The Reliance Hypothesis: Chain-of-Thought Faithfulness Rises as Language Resource Level Falls
Computation and LanguageArtificial Intelligence
Summary
Large language models explain their answers by showing their reasoning, but this is mostly studied in English. The authors found that in languages with less training data, models rely more on reasoning when they don’t know the answer directly, especially for questions needing factual knowledge. This pattern doesn’t hold for questions requiring deeper reasoning. They call this the Reliance Hypothesis and show it across different models and languages.
What this means in practice
- •For ai developers: Improve monitoring tools for verifying model reasoning in low-resource languages to increase safety and trust in multilingual AI systems.
- •For localization teams: Tailor language model evaluation strategies by focusing more on reasoning explanations in languages with limited training data for knowledge-focused tasks.
Authors
Lucas Resck, Jingyi Sun, Qianli Wang, Isabelle Augenstein, Anna Korhonen
Abstract
Chain-of-Thought (CoT) faithfulness matters for monitoring, explaining and safely deploying large language models, yet it has been studied mostly in English. Languages differ in level, the amount of text available to learn from, so models know less in lower-resource languages, which may change how much they rely on their reasoning. To our knowledge, we are the first to relate CoT faithfulness to both resource level and task type (knowledge- or reasoning-intensive). On parallel multiple-choice questions from Multilingual Massive Multitask Language Understanding (MMMLU) in eight languages, faithfulness rises as resource level falls on knowledge-intensive tasks but not on reasoning-intensive ones, a gap that points the same way across four models and several faithfulness measures. We propose, and find support for, the Reliance Hypothesis: models rely more on their reasoning when they cannot recall the answer, as in lower-resource languages on knowledge-intensive tasks. Model confidence partly explains the effect of resource level, but not the knowledge-reasoning gap. Languages in which models are less confident rely more on reasoning, yet less confident answers within a language do not, a Simpson's paradox. In lower-resource languages, models rely on their reasoning for more answers, not more strongly for each. CoT monitoring may therefore be most informative in lower-resource languages, where it is least often validated.