Musec optimizer improves training stability for large language models
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Machine Learning
Summary
Training large language models can be tricky because sometimes the model training becomes unstable and the model's parameters blow up, causing training to fail. The authors propose Musec, a new method that adjusts a specific part of the training process to prevent this instability, making training more reliable without needing special changes to model design. They also develop a smooth version called Soft Musec that works efficiently and maintains good performance. Their results show that Musec helps training stay stable even in tough cases where earlier methods fail.
What this means in practice
- •For machine learning engineers: Train large language models more reliably by using Musec to prevent weight explosions and loss spikes during optimization.
- •For ai infrastructure teams: Deploy training workflows that remain stable at higher learning rates using Musec without needing model-specific modifications.
Authors
Zhuanghua Liu, Menglian Wang, Luo Luo
Abstract
Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. We further develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. To the best of our knowledge, this is the first convergence guarantee for Muon-type methods in the nonconvex nonsmooth setting. We provide empirical studies to show that Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes. Notably, Soft Musec remains stable in settings where existing Muon variants diverge, while matching their performance under well-tuned configurations.