AI summaryⓘ
The authors studied a method called On-Policy Distillation (OPD), which uses a teacher model to guide a student model at a detailed token level. They found that the teacher's guidance is noisy and that the student model improves even when this noise is removed, suggesting OPD mainly helps by lowering the chance of rare tokens rather than by teaching from the teacher. To build on this, the authors proposed a new method called On-Policy Self-Adaptation (OPSA), which does not rely on a teacher and instead uses adaptive signals to suppress unlikely tokens and boost common ones. OPSA showed much better results in experiments compared to both the base model and OPD. Their work shows that effective learning can happen without teacher supervision and that focusing on token probabilities is key.
On-Policy DistillationReinforcement Learning with Verifiable Rewards (RLVR)Token-level SupervisionTeacher-Student ModelsEntropy-adaptive AdvantagesNegative AdvantageLow Log-Probability TokensModel DistillationPass@32Language Model Training
Abstract
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.