Teacher advantage in on-policy distillation lies in targeted corrections
Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?
Machine Learning
Summary
Training smaller models to learn from bigger, smarter 'teacher' models can save time, but it's unclear what exactly the smaller models learn from the teachers. This paper studies where the teacher helps most by flipping mistakes in the student's decisions and measuring gains. They find that carefully chosen corrections help a lot more than random ones, and that training should focus on learning these key fixes while keeping stable behavior. Their method also helps evaluate how much students pick up teacher preferences beyond just accuracy.
What this means in practice
- •For machine learning engineers: Improve student model training schedules by focusing on valuable teacher corrections to achieve closer teacher-level performance.
- •For ai model evaluators: Use controlled token flips and fork-resolution analysis to evaluate what teacher preferences a student model has learned beyond accuracy.
Authors
Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An, Wan Du
Abstract
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.