Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
2026-08-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors found that when large language models seem to fail reasoning tasks, it might not be because they lack reasoning skills but because the way their answers are generated fails at the final step. By looking inside the model’s hidden states, they could still find correct answers even when the model's usual output ranking was wrong. They created a simple correction method that improves accuracy significantly by adjusting output without needing labeled data, showing that the models often have the right logic internally but struggle to express it properly. This suggests that evaluation tests might underestimate a model's true reasoning ability.
large language modelsreasoning taskshidden-state probesoutput bottlenecksequence scoringdiagnostic protocolzero-shot reasoninglexical overlappermutation baselinemodel evaluation
Authors
Qiyao Yan, Chenpeng Wang, Liangming Pan
Abstract
When a large language model fails a reasoning task, it is often assumed to lack the underlying capability. However, this conflates a genuine absence of reasoning with a late-stage output bottleneck. We observe a consistent readout gap across diverse reasoning benchmarks: hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses due to structural biases. To test whether instance-specific logic survives this collapse, we introduce a diagnostic protocol using a minimal, target-label-free additive correction. Fitting just two parameters on as few as 25 unlabeled examples recovers 9--34 accuracy points for Qwen3.5 models, transferring successfully to OLMo-2-1B and Llama-3.1-8B. Crucially, these recovered decisions persist on hard instances unresolved by simple lexical overlap and significantly exceed count-preserving permutation baselines. Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic, urging a narrower interpretation of benchmark evaluations.