MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
2026-06-29 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors address the challenge of combining multiple skills into a single large language model after initial training. They propose a method called Multi-teacher On-Policy Distillation (MOPD), which first trains separate expert models for different tasks and then teaches these skills to one model using its own experience. This approach avoids common problems like inefficient training and losing skill performance. Their method outperforms existing techniques and allows experts to be developed separately without interfering with each other. They successfully tested MOPD on a large model and used it in a real-world industrial setting.
large language modelsreinforcement learningpost-trainingdistillationexposure biasmulti-domain learningon-policy learningmodel fine-tuningQwen3-30BMiMo-V2-Flash
Authors
Wenhan Ma, Jianyu Wei, Liang Zhao, Hailin Zhang, Bangjun Xiao, Lei Li, Qibin Yang, Bofei Gao, Yudong Wang, Rang Li, Jinhao Dong, Zhifang Sui, Fuli Luo
Abstract
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.