On-policy distillation amplifies teacher feedback but risks repetitive outputs
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Artificial IntelligenceComputation and Language
Summary
When teaching a smaller language model by copying a larger one’s behavior, sometimes the smaller model gets better at generating correct answers. But other times, it starts producing long strings of repeated words, even if the teacher model rarely does this. The paper explains this happens because the teacher’s feedback unintentionally encourages such repetitive behavior. The researchers found ways to reduce this problem by ignoring bad responses during training and starting from a well-trained model.
What this means in practice
- •For language model developers: Improve language models’ output quality by preventing repetitive text collapse during on-policy distillation training.
- •For chatbot engineers: Use masking and initialization techniques during training to ensure more consistent and high-quality responses in conversational agents.
Authors
Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang
Abstract
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.