Four rewards improve model reliance on reasoning despite accuracy trade offs
Chain of Faith: Policy-Internal Rewards for Reasoning Reliance
Artificial IntelligenceMachine Learning
Summary
Language models sometimes give answers that sound reasonable but don’t actually depend on the reasoning they show. This paper looks at four ways to reward a model when it truly relies on its reasoning to reach an answer. The authors tested these reward methods on smaller models doing math and strategy questions, finding that some methods make models rely more on their reasoning steps, though it can affect how accurate they are. They also found that high reward settings can cause the model to just copy answers and that there’s a balance between trusting reasoning and getting the right answer.
What this means in practice
- •For machine learning engineers: Improve language model training to increase dependence on reasoning steps when generating answers in math and strategy tasks.
- •For natural language processing practitioners: Develop more robust reasoning generation in small instruction-tuned models through reward signals promoting reasoning reliance.
Authors
Fanyu meng, Zhaodan Kong, Xin Liu
Abstract
Chain-of-thought (CoT) explanations can sound plausible while having little influence on a model's final answer. We study four rewards that use the policy's own signals to encourage reliance on generated reasoning: gradient sensitivity, confidence gaps, progressive confidence gains, and reasoning--prompt swaps. We add each reward to group relative policy optimization (GRPO) and evaluate two small instruction-tuned models on GSM8K and StrategyQA using reasoning edits, paraphrases, and prompt hints. Several configurations increase answer reliance to reasoning edits, while effects on accuracy and paraphrase robustness vary. We also observe answer copying at a high reward weight and high reliance in decomposition settings with poor task accuracy. The study compares practical objectives for reasoning reliance, confirms the trade-off between reliance and performance, and proposes four lightweight rewards that can improve reasoning reliance.