Chain of thought traces often show wrong reasoning despite correct answers
Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces
Computation and LanguageArtificial Intelligence
Summary
Chain-of-thought traces are supposed to show how AI models think when solving problems, helping people check and understand their reasoning steps. The authors tested this on a math problem set where the correct steps are known exactly. They found that many times models get the right answer but with flawed or invalid reasoning traces, especially on harder problems. Even training with shuffled or altered traces still led to good answers, which means traces don’t always reflect real reasoning. This challenges how much we can trust these traces to explain AI thinking.
What this means in practice
- •For ai system developers: Improve debugging and auditing tools by recognizing that correct answers may come with invalid reasoning traces, highlighting the need for better trace verification techniques.
- •For automation engineers: Design AI workflows that do not overly rely on natural-language reasoning traces as evidence of correct internal planning or problem-solving steps.
Authors
Ratish Puduppully, Pranabendu Misra, Paarth Iyer, Durgesh Kalwar, Vardhan Palod, Subbarao Kambhampati
Abstract
Chain-of-thought traces are widely read as records of how models reach their answers, informing debugging, agent auditing, and claims about reasoning. Testing this interpretation is difficult because natural-language thinking traces are rarely mechanically verifiable. We revisit it in iGSM, a synthetic grade-school mathematics benchmark designed to study thinking traces and used to support claims of learned reasoning and planning. Crucially, iGSM exposes the exact quantities and dependencies that a correct solution should use, allowing generated traces to be checked programmatically step by step and enabling us to test whether correct answers are reliably accompanied by valid traces. We first evaluate models trained exclusively on valid, minimal traces. Answer correctness and trace validity nearly coincide in distribution but decouple out of distribution: on the hardest instances, 31.6% of correct answers have invalid traces, over half of which pass all syntactic and arithmetic checks but fail semantic dependency checks. We then intervene on trace supervision. Non-minimal training traces induce non-minimal outputs, while re-asking the same problem with a different query reveals computations inherited from the original query, weakening minimality as evidence of selective planning. Shuffling tokens in 10% of training trace sentences preserves near-clean accuracy even out of distribution despite no trace passing verification. Swapped training traces likewise retain high in-distribution accuracy. We discuss the implications of these findings for chain-of-thought monitoring and interpretation in the context of AI safety.