Student-conditioned updates improve teacher performance in on-policy distillation

On the Off-Policy Teacher in On-Policy Distillation

Machine LearningArtificial IntelligenceComputation and Language

Summary

This paper addresses a challenge when teaching AI models called on-policy distillation. In this process, the student model learns from its own experiences but is guided by a teacher model that was trained on different behaviors. The authors find that the teacher struggles to guide the student well as the student’s behavior differs more from what it originally learned. They propose a method called SCOUT, where the teacher is periodically retrained to better handle the student’s behavior. This approach helps the teacher give better guidance, improving the student’s learning across different setups.

What this means in practice

Authors

Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Blöbaum, Purak Jain

Abstract

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.