Rc-opd improves reasoning in ai by fixing root errors early
Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation
Computation and Language
Summary
When AI systems solve problems step by step, they sometimes make mistakes but don’t learn why those mistakes happened. The authors propose a method called Root-Cause-Guided On-Policy Distillation (RC-OPD) that finds the earliest error in the AI’s steps and helps fix it before moving forward. This approach guides the AI more precisely, improving its reasoning accuracy over time. Experiments showed that this method helps AI models perform better by fixing specific errors instead of just copying correct answers.
What this means in practice
- •For ai model developers: Improve training efficiency and reasoning accuracy of AI by guiding models to fix specific reasoning errors during learning.
- •For automated decision system engineers: Increase reliability of automated reasoning systems in complex domains by integrating targeted repair and feedback to fix reasoning failures.
Authors
Chenglei Shen, Haoyang Yao, Weijie Yu, Song Jin, Xiao Zhang, Jun Xu
Abstract
On-policy self-distillation (OPSD) uses reference solutions as privileged hindsight to supervise student-generated reasoning trajectories. However, reference-based guidance may explain a correct solution without addressing why the student's own reasoning fails. This reasoning mismatch between the guidance provided and the correction needed can encourage the student to borrow correct conclusions while leaving its reasoning errors unresolved. Moreover, applying the same hindsight throughout the trajectory risks a distillation trap, where unnecessary constraints on valid reasoning compete with correction of substantive errors. To address these issues, we propose Root-Cause-Guided On-Policy Distillation (RC-OPD), which uses repairs of the student's own reasoning to provide guidance that addresses its specific errors while building on valid progress. For each failed attempt, RC-OPD locates the earliest substantive error, develops a local correction, and uses the corrected intermediate result as an anchor for the valid prefix. An iterative diagnosis--repair--continuation process tests the repairs through student continuation, identifying further errors within a fixed repair budget. For repair chains that reach a correct answer, root--cause--guided distillation uses failure diagnoses and corrective goals to supervise the erroneous segments, while anchor-guided distillation supports the corresponding valid prefixes with reasoning chains leading to the repaired intermediate results. We evaluate RC-OPD across multiple datasets and model scales. Extensive experiments and analyses show that it mitigates reasoning mismatch and the distillation trap, yielding substantial performance gains.