Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study a method called On-Policy Self-Distillation (OPSD), where a 'teacher' model learns by looking at the correct answer and guiding a 'student' model. They question if the improvements come only from seeing the correct answer or from other factors. To explore this, they try a new approach, OP²SD, which uses solutions from different problems instead of the exact correct answer. They find that OP²SD still helps the model improve, suggesting that the teacher's way of guiding matters more than just having the exact answer.
On-Policy Self-Distillationteacher-student learningmodel distillationtoken-level supervisionreference solutionmodel training contextmathematics benchmarks
Authors
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar, Junpei Komiyama
Abstract
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with $\mathrm{OP}^{2}\mathrm{SD}$ (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, $\mathrm{OP}^{2}\mathrm{SD}$ improves over the base model, remains competitive with OPSD. The success of $\mathrm{OP}^{2}\mathrm{SD}$ implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.