Muon training chooses Frobenius margin over spectral due to practical tweaks
From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
Machine LearningArtificial Intelligence
Summary
This paper looks at how a training method called Muon behaves differently in theory versus practice. Theoretically, Muon focuses on a property called spectral max-margin, but when used with real-world approximations, it actually picks a different property called Frobenius max-margin. The authors prove this switch happens because of specific approximation steps in Muon's implementation. They also describe when Muon can go back to its original behavior by increasing certain computational steps gradually during training.
What this means in practice
- •For machine learning engineers: Design training algorithms that predict when practical approximations might switch learning biases in models using Muon or related methods.
- •For numerical optimization developers: Develop orthogonalization routines aware of how iteration counts and stabilization affect long-term optimization biases for better algorithm stability.
A theory result. No direct application yet.
Authors
Yongtao Wu, Mahdi Mansouri, Zhenyu Zhu
Abstract
Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.