Know When to Stop: Segment-Level Credit Assignment for Reducing Overthinking
2026-07-01 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how reasoning language models sometimes get stuck by overthinking and producing confusing or wrong answers, even when the length of their responses is the same. They found that paying attention to intermediate answers during reasoning helps identify when the model is improving or getting worse without needing costly manual checks. They developed a method called DASH that gives feedback only when the model’s steps move closer to the correct answer. This approach improved performance on challenging math tests and reduced useless overthinking.
reasoning language modelsoverthinkingself-reflectionintermediate answer commitmentscredit assignmentDASHmath benchmarksAIMEself-correctionresponse length control
Authors
Chia-Hsuan Lee, Sihui Dai, Mingyang Zhou, Isha Slavin, Shi-Xiong Zhang, Sambit Sahu, William Campbell
Abstract
Reasoning language models frequently overthink: generating extended chains of behaviors such as hedging, approach abandonment, and self contradiction that consume tokens without improving answers. We show that these behaviors are not merely a consequence of length; even when controlling for response length, incorrect traces exhibit higher rates of unproductive self-reflection than correct ones. Addressing this requires identifying where self-reflection helps vs hurts, but obtaining these step-level annotations is costly. We observe that intermediate answer commitments within reasoning traces can provide a cheap proxy: by comparing each final answer candidate in the trace to the ground truth, we can determine whether subsequent reflection is productive without any additional supervision. Building on this insight, we propose DASH (Drift Aware advantage SHaping), which assigns segment-level credit based on whether each reasoning segment leads toward or away from correctness. On competition-level math benchmarks, DASH achieves the highest accuracy where overthinking is prevalent (AIME25: 50.8% vs. 45.4% GRPO) while reducing overthinking behaviors and achieving more productive self-correction than baselines.