Row normalization affects training speed of large language models

The Row Normalization Puzzle in Muon

Machine Learning

Summary

This paper looks into how a technique called row normalization influences the training of large language models, like the ones used in AI. The authors found that row normalization can slow down the process in some worst-case scenarios but still helps in practice during real training. They proved mathematically why this slowdown happens and showed through experiments that despite this, the method works better on actual language tasks. This raises questions about why it helps in real-world cases even though theory suggests it should not always be faster.

What this means in practice

  • For machine learning engineers: Optimize large language model training by understanding when row normalization speeds up or slows down learning processes.
  • For ai model trainers: Choose training techniques for language models with clearer insight into trade-offs of row normalization effects on convergence.

Authors

Jiayu Zhang, Tianyi Lin

Abstract

This paper examines how row-wise renormalization affects Muon, focusing on the gap between NorMuon's worst-case guarantees and its practical performance (Li et al.). Despite its growing adoption and promising performance in large language model (LLM) pretraining, NorMuon's worst-case guarantees remain poorly understood. One fundamental question is: Does row normalization yield provable convergence gains, potentially through its interaction with approximate polar computation and exponential moving-average momentum? Our results show that row normalization introduces a dimension-dependent factor in the worst-case iteration complexity under the operator-norm geometry, which persists even with exact polar computation and any fixed momentum parameters. Indeed, we establish an algorithm-dependent lower bound and a matching upper bound in deterministic settings, and extend our upper bound analysis to stochastic settings. Both upper-bound analyses allow approximate polar computation. Experiments show that NorMuon is slower than Muon on synthetic problems inspired by our worst-case construction, yet outperforms Muon in LLM pretraining. These findings sharpen the puzzle of why row normalization helps in practice and complement the recent findings of Dewulf et al.