Convergence of practical muon improves training efficiency in neural networks
Convergence of Practical Muon
Machine Learning
Summary
Training large neural networks often uses a method called AdamW, but another technique named Muon shows promise. The authors studied Muon's real-world version, which includes certain practical steps often ignored in theory. They showed that this practical Muon approach will reliably get closer to the best solution when training, and it does so more efficiently than AdamW in high-dimensional settings. Their experiments support these findings, helping explain why Muon works well in practice.
What this means in practice
- •For machine learning engineers: Use practical Muon to train large neural networks with improved convergence rates over AdamW.
- •For ai infrastructure teams: Improve training stability and efficiency in deep learning frameworks by integrating Muon optimizer with tuned polynomial parameters and decoupled weight decay.
Authors
Haonan Wang, Yu Wu, Minghui Liwang, Xinlei Yi, Yiguang Hong
Abstract
Muon is emerging as a promising alternative to AdamW for large-scale neural network training, yet theoretical understanding of its practical implementation remains incomplete, as existing analyses often simplify or omit two key components: (i) practical Newton--Schulz iterations with empirically tuned polynomial coefficients $(3.4445,-4.7750,2.0315)$; and (ii) decoupled weight decay for regularization. In this paper, we provide an optimization interpretation and establish convergence for practical Muon, jointly accounting for both components. Specifically, we interpret practical Muon as right-preconditioned optimization of the original loss with a dynamic weighted $\ell_2$ regularizer that vanishes as stationarity is approached, so that the optimization target remains the original objective. We then establish, to our best knowledge, the first convergence guarantee for practical Muon in the stochastic nonconvex setting, with an $\mathcal{O}(T^{-1/4})$ convergence rate in terms of the expected Frobenius norm of the gradient, improving the dimension dependence of the best known AdamW's convergence rate by a factor of $\sqrt{d}$, where $T$ is the iteration horizon and $d$ is the parameter dimension. Experiments further support the theoretical convergence results.