Muon optimizer makes first mixture-of-experts layer harder to balance
Muon Makes the First Mixture-of-Experts Layer Harder to Balance
Machine LearningComputation and Language
Summary
Training large language models with mixture-of-experts layers involves balancing workload across different parts called experts. The authors found that using the Muon optimizer tends to increase imbalance in the very first layer, making some experts work much more than others. They showed that adjusting input values or lowering Muon's learning rate can reduce this problem without hurting performance. Their results suggest it’s better to keep the router training with Muon rather than switching optimizers at this scale.
What this means in practice
- •For machine learning engineers: Improve training stability and load balancing in MoE language models by adjusting Muon optimizer settings and input normalization approaches.
- •For ai model trainers: Optimize expert usage in natural language models by selectively tuning optimizer parameters for mix-of-experts layers during warmup phases.
Authors
Marcos Hernanz
Abstract
Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.