Multi teacher approach improves training of large language models
Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation
Computation and Language
Summary
Teaching a large language model is tricky because different parts of its responses might need different guidance. The authors found that instead of using one teacher for an entire response, it’s better if each word in the response can choose its own teacher. They created a new method where multiple teachers help the model, selecting the best one for each word during training. This approach made the models better at solving math problems in tests.
What this means in practice
- •For language model engineers: Improve fine-tuning accuracy of language models by adaptively choosing multiple teachers for each token during training.
- •For ai service providers: Enhance the performance of large language models in mathematical reasoning tasks by applying token-level adaptive self-distillation.$Commercial implications: Enables production-quality AI services with more accurate model outputs, improving competitive advantage in AI-assisted reasoning tools.
Authors
Yifu Luo, Chuang Ma, Zeyu Chen, Enkelejda Kasneci, Qi Chen, Gjergji Kasneci
Abstract
On-policy Self-distillation (OPSD) has proven effective for post-training large language models by constructing a teacher conditioned on privileged references. However, existing approaches keep the teacher fixed either throughout training by consistently exposing it to the full reference, or within each response by adaptively varying partial reference exposure only across responses. In this paper, we argue that neither of them is sufficient. Instead, every token needs its own teacher, as the teacher preference can vary token by token within the same response. Through a teacher-intervention pilot study, we validate that the teacher preference differs across tokens, and reveal that a preferred teacher can be identified for each token through majority voting and KL-based selection. Motivated by these findings, we propose Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD), the first token-level adaptive OPSD method. Our method constructs a multi-teacher pool consisting of candidate teachers, and enables every token to adaptively select its preferred teacher during training. Experiments with Qwen3 models on mathematical reasoning tasks show that MTA-OPSD consistently outperforms existing OPSD approaches with an average gain of up to $+5.9$ over the base model, highlighting the importance of adapting teacher supervision at the token level. The code is available at https://github.com/xingzhejun/MTA-OPSD.