SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
2026-06-08 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors studied a method called on-policy distillation (OPD), where a student model learns by following its own steps but using guidance from a stronger teacher. They found that OPD assumes the student and teacher behave very similarly and that the teacher's advice is always reliable, which is often not true. To fix this, they created Sign-Gated On-Policy Distillation (SG-OPD), which uses a verifier to check when to trust the teacher's guidance and adjusts learning accordingly. Their tests on math problem benchmarks showed SG-OPD performs better than the original OPD method.
on-policy distillationoff-policy distillationreinforcement learningtrajectory alignmenttoken-level supervisionteacher-student modelsbinary verifiersign-consistency gatemathematical reasoning benchmarks
Authors
Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, Xiaosong Yuan
Abstract
On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.