MuonGS improves language model training efficiency with relaxed norm control
MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
Machine LearningArtificial Intelligence
Summary
Training large language models involves carefully controlling how much the model’s weights change during learning. The authors study a measure called accumulated effective learning rate to understand why and how certain controls help. They propose MuonGS, a new method that adjusts updates without fixing the weight size, allowing more effective training progress over time. Their experiments show MuonGS lowers validation errors and speeds up training compared to previous methods.
What this means in practice
- •For machine learning engineers: Train large language models more efficiently by using MuonGS to control weight updates for faster convergence and better validation results.
- •For ai infrastructure teams: Improve resource usage and training speed in distributed systems by adopting MuonGS optimizer for large-scale language model pretraining.
Authors
Athanasios Glentis, Dawei Li, Mingyi Hong
Abstract
Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We analyze norm control in Muon over the entire training trajectory using the \emph{accumulated effective learning rate}, $C_T=\sum_{t<T}η_t/\|W_t\|_F$, where $η_t$ is the nominal learning rate and $η_t/\|W_t\|_F$ is the effective learning rate relative to the weight norm. For bounded update directions, accumulation over any training interval bounds relative weight displacement. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Going beyond bounded-norm control, we show that sufficiently slow norm growth guarantees faster accumulation than the logarithmic worst case, opening up less restrictive optimizer designs. Based on this insight, we propose MuonGS, which uses Gram--Schmidt (GS) projection to remove the radial component of each Muon update without fixing the weight norm. With bounded update directions and a constant nominal learning rate, we prove that the weight norm grows at most proportionally to $\sqrt{t}$ after $t$ steps. Consequently, MuonGS guarantees $C_T=Ω(\sqrt{T})$, with accumulation sustained even in later training. Further, for smooth scale-invariant objectives we establish its $O(T^{-1/2})$ stationarity guarantee. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons. In particular, for the dense model it achieves an estimated token-equivalent speedup up to about $15\%$ at matched validation loss to MuonH.