AI summaryⓘ
The authors studied how on-policy distillation (OPD), a method where a student model learns from a teacher model using the student's own behavior, generalizes beyond its training data. They found that OPD mostly transfers the way the teacher thinks rather than just copying correct answers, and this works even with hard problems the teacher can't solve. The success of OPD depends a lot on how similar the teacher and student models are, with closely related pairs generalizing well across different tasks and domains. However, using multiple teachers can cause mixed results because their influences overlap unpredictably. These insights help understand when and how OPD works best, especially with multiple teachers.
on-policy distillationteacher-student modelsgeneralizationpolicy learningcross-domain transfermulti-teacher learningtrajectory samplingmodel reasoningdistribution shiftmachine learning
Authors
Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, Defu Lian
Abstract
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.