Multi-teacher token-level supervision improves on-policy distillation
MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Machine LearningArtificial Intelligence
Summary
Teaching an AI student using multiple expert teachers usually assigns one teacher per task domain for the whole learning session. This method limits flexibility and wastes useful advice from other teachers. The authors propose a system that chooses the best teacher advice token by token, without needing to know the domain in advance or train extra models. Their approach, called MOPD-Router with ExpertAlign scoring, consistently improves the student AI’s performance across different setups and unlabeled data. This shows more detailed teacher guidance can better combine expert knowledge from many sources.
What this means in practice
- •For natural language processing teams: Create more adaptable AI language models by integrating diverse expert feedback without relying on predefined domain labels.
- •For ai service providers: Improve multi-domain AI assistants’ performance by dynamically weighting teacher signals to reduce dependence on labeled prompt data.$Commercial implications: Enables selling smarter AI assistants that adapt across domains without manual domain tagging, reducing deployment complexity.
Authors
Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
Abstract
Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher-student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment. Code is available at: https://github.com/TURLEing/MOPD-Router.