Interactive-policy distillation improves training efficiency and accuracy in student ai models
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Machine LearningArtificial Intelligence
Summary
Training smaller AI models to think like bigger expert models can be tricky because the smaller ones sometimes drift away from the expert guidance. The authors developed a new method where the student and teacher models take turns suggesting and checking parts of the reasoning process. This back-and-forth approach helps the smaller model learn more accurately and faster. They tested it on difficult math problems and found it needed less data and fewer training steps to get better results compared to older methods.
What this means in practice
- •For ai model developers: Improve efficiency and accuracy of training smaller language models by using interactive bidirectional teacher-student supervision.
- •For machine learning platform engineers: Build inference systems that host teacher and student models jointly to speed up dual-model interactive training.
Authors
Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
Abstract
On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.