Muon spectral optimizers show three distinct behaviors on simple quadratic problems
Constant Step Sizes for Muon-Type Spectral Optimizers on Strongly Convex Quadratics
Machine Learning
Summary
Optimizers named Muon use special matrix transformations to update parameters in machine learning. This work studies how constant step sizes affect their convergence when applied to simple, well-understood problems called strongly convex quadratics. The authors find that depending on how the update scales and normalizes, Muon-type optimizers behave in three different ways: some converge quickly like normal gradient descent, some still converge but differently, and some fail to converge almost always. These findings help explain why different versions of Muon behave the way they do in practice.
What this means in practice
- •For machine learning engineers: Choose stable and efficient constant step sizes for spectral optimizers when training convex quadratic models to ensure reliable convergence behavior.
- •For numerical optimization developers: Design spectral matrix-based update rules with guaranteed convergence properties by selecting normalization and scaling methods informed by this classification.
Authors
Pratik Jawanpuria, Bhabatosh Garai, Bamdev Mishra
Abstract
Muon and related optimizers update a weight matrix with a spectrally transformed gradient. In this work, we study such spectral optimizers with a constant step size on strongly convex matrix quadratics, which isolate the effect of the update from the confounds of nonconvexity, noise, and step size schedules. We analyze an update using two quantities: the gain, which is the factor by which it amplifies the gradient along each singular direction, and the length, which is the norm of the update. These two quantities place the popular spectral updates in three regimes with different behavior: (i) with a bounded gain, the iterates behave like preconditioned gradient descent and converge for step sizes set by the largest curvature; (ii) with an unbounded gain but a vanishing length, they still converge, with the admissible step sizes set by the total curvature instead; and (iii) with a non-vanishing length, they do not converge for almost every initialization. Interestingly, the practical variants of Muon fall in all three regimes, depending only on how the update is normalized and scaled and not on the polynomial that approximates the polar factor. Experiments validate these findings.