AI summaryⓘ
The authors study a way to help AI models solve math problems better by teaching them using extra hints from the solution, a technique called on-policy self-distillation (OPSD). They found that directly using the full solution as a guide can give the student model information it won't have when actually solving problems. To fix this, they introduced Problem-Space-Guided OPSD (PS-OPSD), which gives the student model more general guidance about the problem and its steps instead of the full solution. Their tests on different math benchmarks showed that this approach helps models perform better when only given the question. The authors also showed that how well the guidance matches the problem and follows a clear step-by-step path matters for improving performance.
On-policy self-distillationMachine reasoningMathematical problem solvingTeacher-student trainingTrajectory-grounded guidanceModel supervisionState-transition pathBenchmark evaluationAI model generalization
Authors
Xuyang Zhao, Liting Zhang, Zichen Xu, Zhihu Wang, Xu Caiyue, Shiwan Zhao, Qicheng Li
Abstract
On-policy self-distillation (OPSD) improves reasoning by using a privileged view of a model conditioned on reference solutions to supervise a student view that observes only the question. However, the teacher-provided token-level targets may depend on reference-specific information unavailable at inference time. We propose Problem-Space-Guided OPSD (PS-OPSD), which replaces the complete solution with trajectory-grounded guidance describing the initial state, goal conditions, constraints, and a selected state-transition path. The student rollout and OPSD objective remain unchanged. Across three mathematical reasoning benchmarks and model scales ranging from 1.7B to 8B, PS-OPSD achieves the highest aggregate question-only accuracy among the compared methods. Controlled experiments further indicate that guidance relevance and path coherence contribute to these gains, highlighting the representation of privileged information as an important design choice in OPSD.