Denoise First, Orthogonalize Later: Understanding Momentum in Muon via Spectral Filtering
2026-06-02 • Machine Learning
Machine Learning
AI summaryⓘ
The authors explore why adding momentum helps improve the Muon optimizer used in training large language models. They show that momentum acts like a filter that reduces noise (perturbations) in the training signals while keeping the main useful information. This filtering makes the model updates more stable and reliable, especially when done before a key step called orthogonalization. Their theoretical findings are backed up by experiments, providing insight into how momentum benefits matrix-based optimization methods.
Muon optimizermomentumspectral filtergradient modelspectral gapsingular subspacesorthogonalizationlarge language modelsmatrix-based optimizers
Authors
Xianliang Li, Zihan Zhang, Weiyang Liu, Han Bao
Abstract
Muon has recently demonstrated strong empirical performance in large language model training, but the theoretical role of momentum in Muon remains unclear. Existing analyses of Muon either remove momentum to study spectral updates in isolation, or retain momentum without explaining why it improves empirical performance. Our work bridges this gap by showing momentum in Muon acts as a spectral filter. Under a structured signal-plus-perturbation gradient model, we prove that momentum suppresses perturbations while preserving the dominant signal, thereby enlarging the spectral gap between them. This enlarged gap stabilizes the singular subspaces of the matrix passed to Muon's orthogonalization step, making the resulting update more reliable. We further show that applying momentum before orthogonalization achieves provably stronger alignment with the signal component of the gradient than either reversing this order or simply removing momentum. Experiments across diverse tasks, including LLM pretraining, support our theoretical analysis. More broadly, our theory offers a starting point for understanding the benefits of momentum in other matrix-based optimizers.