Constant-step momentum improves stability in stochastic gradient descent

A Nonlinear Time-Series Theory for Constant-step SGD with Momentum

Machine Learning

Summary

Stochastic gradient descent with momentum is a common method for training machine learning models by updating parameters while remembering past updates. The authors analyze this method as a kind of nonlinear time series to better understand its long-term behavior and stability. They show the method steadily forgets initial starting points under certain conditions, leading to predictable, stable behavior that resembles a Gaussian process when averaged or in the limit of small learning steps. This gives new insights into why momentum helps training and how it affects variability in the updates.

What this means in practice

A theory result. No direct application yet.

Authors

Younghun Lee, Jake Zummo, Wei Biao Wu

Abstract

Constant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite $p$th moment for $p > 1$, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.