Calibrating teacher student differences improves on policy distillation
Calibrating Teacher--Student Discrepancy for On-Policy Distillation
Artificial Intelligence
Summary
When a smaller model (student) learns from a bigger, smarter model (teacher), sometimes the differences they notice include teacher quirks, not just what the student needs to learn. The authors found a way to separate the teacher's own unusual behavior from the real differences the student should focus on. Their new method, Cal-OPD, keeps only the meaningful differences for training. Tests with math problem-solving show this leads to better student performance than previous techniques.
What this means in practice
- •For machine learning engineers: Improve training of reasoning models by filtering out teacher-specific noise in teacher-student learning setups.
- •For ai system developers: Enhance model distillation processes in products relying on on-policy student training to achieve better performance.
Authors
Qiangqiang He, Jin Li, MingCai Chen
Abstract
On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce \textbf{Calibrated On-Policy Distillation (Cal-OPD)}, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.