On-policy self-distillation shows mixed results improving reasoning models
Diagnosing On-Policy Self-Distillation for Reasoning Language Models
Computation and Language
Summary
Improving language models to do better math reasoning is tricky. The authors studied a method called on-policy self-distillation, where a model teaches itself without outside help. They found that this method only helps under very specific conditions and can even make the model worse otherwise. The study shows this approach is quite sensitive and not a reliable way to improve reasoning in language models.
What this means in practice
- •For language model engineers: Avoid relying on on-policy self-distillation to boost reasoning performance unless reasoning modes between teacher and student align closely.
- •For ai safety teams: Recognize potential risks of behavior collapse when deploying self-distillation techniques in reasoning language models.
Authors
Yang Li, Gongle Xue, Yuheng Yuan, Yijia Guo, Shizhe Zhang, Liwen Hu, Lei Ma
Abstract
On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B--8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.