Adaptive iterative repair improves on-policy distillation for math reasoning models
Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation
Machine LearningComputation and Language
Summary
The paper focuses on teaching AI models to fix their own mistakes when solving math problems by giving detailed feedback on each error. The authors developed a method called AIR-OPD, which helps the model get better by guiding it step-by-step through its errors and retries. This approach uses a teacher model that provides hints based on the current mistake, allowing the student model to try again. They tested it on various math test sets and found it improved the models' problem-solving skills without hurting performance on unrelated tasks.
What this means in practice
- •For natural language processing teams: Improve AI models that solve math problems by enabling them to iteratively fix errors using adaptive guidance.
- •For automated customer support developers: Enhance conversational agents with better error recovery by applying stepwise error repair techniques from on-policy distillation.
Authors
Rui Li, Liyang He, Zheng Zhang, Zhenya Huang, Linbo Zhu, Qi Liu
Abstract
On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal than the outcome-level rewards of reinforcement learning. This feedback comes from a teacher conditioned on a full reference solution unavailable to the student. The reference solution specifies the target but not how to move from the student's current error toward it, creating a solution-conditioned shortcut risk. We introduce AIR-OPD, an adaptive iterative repair framework for on-policy distillation that provides error-to-repair supervision. Given a failed response, a guidance generator synthesizes repair guidance for the current error. The student samples an on-policy retry with this guidance. If the retry remains incorrect, the generator produces new repair guidance for the newly observed error. At each round, a fixed teacher receives the guidance as privileged context and supervises the student on an error-aligned region of its latest failed response. Outcome-aware stage weighting favors early repair stages and credits stages whose immediate retry passes verification. We train AIR-OPD on the DAPO-Math-17K dataset and evaluate on AIME24, AIME25, and HMMT25, alongside out-of-distribution tests on MMLU-Pro and GPQA. We examine two guidance sources, self-guidance from the current student policy and external guidance from a larger model. For both Qwen3-4B and Qwen3-8B, AIR-OPD attains the best mathematical-reasoning averages, improving over the strongest baseline by up to 3.6 points, while preserving base-model performance on the out-of-distribution benchmarks.