Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study why small language models struggle with physics reasoning, finding that one mistake early on causes many later errors. They introduce a new training method that detects the first error, gives focused feedback, and helps the model fix its answer without needing correct solutions as examples. Their approach improves accuracy on physics problems and reduces calculation and understanding mistakes significantly, but conceptual errors remain challenging. This method works better than earlier techniques that require lots of annotated data.
small language modelsphysics reasoningstep-level rewardpolicy gradientKL regularizationchain-of-thought promptingerror propagationmodel trainingconceptual errorscalculation errors
Authors
Raj Jaiswal, Dhruv Jain, Rishabh Dhawan, Sree Krishna Uppalapati, Shin'ichi Satoh, Tanuja Ganu, Rajiv Ratn Shah
Abstract
Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.