Iterative merging improves combining multiple teacher models into one
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Machine LearningArtificial Intelligence
Summary
When combining different expert models into a single one, starting from certain initial combinations is important but hard to predict. The authors found that simply averaging or warming up the student model isn't enough to capture all teachers’ strengths. They propose a method that gradually adds the unique contributions of each teacher during training. This approach leads to better overall performance across different tasks.
What this means in practice
- •For machine learning engineers: Combine multiple specialized AI models into a stronger single model with improved task performance using iterative merging during training.
- •For ai platform developers: Develop tools that merge multiple domain-expert models progressively to enhance efficiency and quality of deployed AI services.$Commercial implications: Enables building commercial AI services that incorporate multiple domain experts into one adaptable model, improving product offerings.
Authors
Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee
Abstract
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.