Finite spectral depth changes early neural net training paths
Finite Spectral Depth Controls Gaussian Entrance Rank and Avoidance of the First Gradient-Flow Saddle
Machine Learning
Summary
Training neural networks often involves slowly changing matrices from simple starting points. This paper studies a method that adjusts these changes using a set number of mathematical steps called Newton-Schulz iterations. The authors found that by changing how many steps are used early on, you can steer the training process to skip certain sticking points and select simpler structures in the matrix. They also proved conditions under which these skips happen and gave an example where just four steps are enough to bypass the usual training pause.
What this means in practice
- •For machine learning engineers: Adjust training algorithms by controlling iteration steps to avoid slowdowns in early matrix learning stages.
- •For optimization algorithm developers: Design continuous-time methods that bypass certain gradients to improve convergence during matrix factorization tasks.
A theory result. No direct application yet.
Authors
Yanlin Long
Abstract
Matrix factorization provides a simple model for studying the training dynamics of neural networks. Recent studies of gradient flow under small initialization have analyzed how trajectories escape from a neighborhood of the origin and approach their first nonzero stationary point. In this paper, we study continuous-time dynamics that applies a finite number of Newton-Schulz iterations to the normalized matrix gradient (Muon). We show that the number of Newton-Schulz steps can change the rank selected during the early stages of training and allow the resulting trajectory to bypass the first nonzero stationary point approached by gradient flow. Under an additional residual-inertia condition, we prove that bypass occurs for all sufficiently large finite Newton-Schulz depths and provide an explicit sufficient threshold. We also construct an example in which the bypass threshold is four.