Small language models improve math reasoning with executable step training
ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
Artificial Intelligence
Summary
Small language models are easier to use locally but struggle with math problems that need precise steps, calculations, and checking. The authors present ARCHER, a method that teaches these models to follow clear, executable reasoning steps using calculator calls and verification. This method trains models in two stages: first by turning free explanations into structured actions, then by refining them through reinforcement learning. Their approach improves math problem-solving accuracy on several benchmarks with models smaller than 10 billion parameters.
What this means in practice
- •For software developers: Improve small AI models for local or privacy-sensitive apps requiring reliable step-by-step math reasoning with integrated tool use and checks.
- •For financial modelers: Use compact language models trained with executable reasoning to automate and verify precise calculations in financial forecasts or audits.
Authors
Dhruv Dixit, Khushi Patel, Congcong Xu, Min Song
Abstract
Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator-grounded \textbf{H}ierarchical \textbf{E}xecutable \textbf{R}easoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.