Optimizer geometry influences how matrix learning unfolds over time
Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization
Machine Learning
Summary
Training methods for matrix factorization can progress differently depending on how they update parameters. The authors study various optimization techniques to understand how they affect the timing and persistence of different features being learned. They find that some methods remove early delays others have in learning smaller components, while others keep or change these delays in complex ways. These findings help explain the role of optimizer design, normalization, and depth in shaping the learning process.
What this means in practice
- •For machine learning engineers: Select optimization methods for matrix factorization tasks to control the emergence and stability of learned features during training.
- •For neural network developers: Design training regimes that manage how different spectral modes appear and fade by tuning optimizer geometry and depth.
A theory result. No direct application yet.
Authors
Mahalakshmi Sabanayagam, Simon Lucey
Abstract
Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-rank solutions by delaying the emergence of small singular modes. This raises the question of what remains of that spectral bias when normalization or curvature correction weakens or removes the slowdown. We study how optimizer geometry and depth jointly govern this behavior through a common framework for singular mode dynamics. Under explicit balance and alignment assumptions, we derive birth, saturation, and decay laws for Euclidean GD, coordinate-wise updates of SignGD and an instantaneous Adam approximation, spectral updates of Muon and cumulative Shampoo, and a block curvature model of K-FAC. The resulting picture is not a simple ordering from stronger to weaker low-rank bias. To highlight, SignGD and ideal Muon eliminate the divergent birth barrier and drive unsupported modes to zero in finite time. Cumulative Shampoo initially retains GD's depth-dependent barrier, then accumulated gradients produce a catch-up phase while making previously active modes increasingly persistent. Undamped K-FAC cancels the factorization-induced slowdown while preserving the ordering of the target singular values, whereas positive damping introduces a spectral threshold below which the slow GD phase laws reappear. These results give normalization, accumulated state, damping, and depth a direct interpretation as controls determining when modes emerge, persist, and disappear during training.