On-policy reverse distillation improves model learning beyond weaker teachers
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Machine LearningComputation and Language
Summary
Sometimes, newer AI models start by learning from older, weaker models but need to get better than them. The authors introduce a method called On-Policy Reverse Distillation (OPRD) that helps newer models learn faster and even surpass their teachers by focusing on rewarding helpful guidance instead of copying everything. This method lets the student model follow its own learning path but still benefit from the teacher’s advice. Tests show OPRD helps models learn more efficiently and perform better in multi-domain and successive learning scenarios.
What this means in practice
- •For machine learning engineers: Train new models faster and achieve better performance by using weaker but available models as guidance without being limited by their quality.
- •For natural language processing developers: Improve multi-domain language models by efficiently consolidating knowledge from multiple weaker models into a stronger student model.
Authors
Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
Abstract
Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduce On-Policy Reverse Distillation (OPRD), which evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Response-style analysis shows that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the student's own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combines verifier-driven policy optimization with teacher guidance regardless of capacity ordering.