Model disagreement reveals errors better than extra compute in reasoning
Not All Test-Time Compute Is Evidence: Model-Specific Bias in LLM Reasoning
Artificial Intelligence
Summary
More computing time on the same language model doesn’t always mean better reasoning because the model tends to repeat the same mistakes. The authors found that when a model’s early and final answers don’t agree, it signals uncertainty. Their method called DIA uses different models to independently answer these tough questions, improving accuracy while using fewer computing resources. This approach worked well on many tests and used less extra effort than other methods involving multiple agents.
What this means in practice
- •For ai developers: Improve reasoning accuracy in language model applications by selectively involving diverse models only when disagreement signals instability.
- •For chatbot engineers: Reduce computational cost in conversation systems by activating extra reasoning only for uncertain responses identified through model disagreement.
Authors
Ming Shen, Tianjun Yao
Abstract
Test-time scaling methods implicitly assume that additional computation from the same model provides independent evidence. We show that this assumption fails within a single reasoning model: errors are correlated across repeated samples, across reasoning depth, and in the model's own self-evaluation, so more compute from the same model is not more evidence. We introduce DIA (Disagreement-triggered Independent Arbitration), which treats disagreement between an intermediate-stage and the final-stage prediction as a per-question signal of reasoning instability and resolves it by asking a small panel of heterogeneous models to solve the question independently, without seeing either candidate, and aggregating their answers directly. Across seven benchmarks and two base models, DIA outperforms full chain-of-thought in 12 of 14 settings, including a 7.00% gain on MuSiQue, while activating on only $\sim 17\%$ of examples and using up to 47.2% fewer tokens than a multi-agent baseline. Ablations show that both independent solving and solver heterogeneity are necessary for these gains.