Moevolve improves mixture of experts models with bidirectional distillation
A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation
Machine LearningArtificial Intelligence
Summary
Large language models often use mixtures of smaller expert models to improve performance, but training them well is hard because feedback can be sparse. The authors show that by temporarily increasing how many experts work together and carefully teaching smaller mixtures using these stronger bigger mixtures, models can learn better. Their method, MoEvolve, lets the smaller model keep its speed but still gain from the extra knowledge of the bigger model. This approach consistently improved model accuracy across many tests, sometimes even beating the bigger models themselves.
What this means in practice
- •For ai model developers: Boost performance of mixture-of-experts language models without increasing inference costs by using MoEvolve's bidirectional distillation.
- •For cloud ai service operators: Improve model accuracy in deployed MoE-based services while maintaining computational efficiency by applying MoEvolve to optimize expert activations.
Authors
Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved Mixture-of-Experts (MoE) LLMs, but its outcome-level rewards provide only sparse supervision. On-Policy Distillation offers dense token-level feedback, yet requires a stronger compatible teacher, while on-policy self-distillation relies on hand-designed privileged information. We observe that the MoE architecture itself offers a way to resolve this dilemma, as increasing the activation budget per token exposes latent expert capacity and yields a potentially stronger teacher within the model. However, naive expansion degrades performance because newly co-activated experts are not optimized to work together. We therefore propose MoEvolve, a bidirectional distillation framework following a recovery-then-improvement strategy: we first restore the activation-expanded high-$k$ model and further strengthen it by exploiting its additional active capacity, then reverse the direction of distillation to transfer the enhanced capabilities back into the original low-$k$ configuration, retaining its inference-time activation budget. Across diverse MoE models and benchmarks, MoEvolve improves the initial models by up to $22.2$ domain-average points and outperforms same-budget RLVR baselines by up to $6.5$ points. Notably, the final low-$k$ models often surpass their activation-expanded teachers.