New distillation method improves cross-family language model training
CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts
Machine Learning
Summary
Teaching smaller language models using larger models works well when both models are similar, but it becomes less effective when they come from different families. The authors found that combining two different influences during training confuses the learning process. They designed a new method called CompassOPD that separates these influences and focuses on the improvements from the better model itself. This approach leads to better results in reasoning tasks across different types of models.
What this means in practice
- •For language model developers: Improve training of smaller models by transferring knowledge effectively from larger, different-family models, enhancing reasoning abilities.
- •For ai system integrators: Use CompassOPD to optimize cross-family model deployment where various model architectures must collaborate or upgrade performance on task-specific problems.
Authors
Naibin Gu, Qingyi Si, Chenxu Yang, Chuanyu Qin, Junhao Zhou, Peng Fu, Zheng Lin, Weiping Wang
Abstract
On-policy distillation (OPD) provides dense token-level supervision on student-generated trajectories. Although OPD performs strongly when teacher and student belong to the same model family, we find that its effectiveness degrades in cross-family settings even after tokenizer alignment, with substantially stronger external teachers offering little additional improvement. To understand this disconnect, we decompose the cross-family OPD signal into two components: an offset between a low-capability teacher-family reference and the student, and the within-family log-likelihood shift from that reference to the strong teacher. Standard OPD transfers both components together, allowing the offset to dominate the update direction and obscure the changes associated with teacher capability improvements. We propose CompassOPD, which removes this offset and transfers the within-family shift, while a frozen student reference anchors updates to the student's initial policy. Thus, both teacher-side and student-side changes are measured within their respective model families. Experiments across three student families and multiple teacher families show that CompassOPD consistently outperforms standard cross-family OPD, improving average reasoning accuracy by up to 5.50 points. For an MoE teacher, we further construct the reference directly from the teacher checkpoint by reducing expert activation, eliminating the need for a separate reference checkpoint while retaining a 3.43-point gain over OPD.