New equations better capture real gradient descent learning dynamics
Stochastic learning flow: moment equations for stochastic gradient descent at finite step size
Machine Learning
Summary
Deep learning often assumes smooth changes in model training, ignoring the effects of step sizes and randomness in updates. The authors derived new equations that model how parameters and their variations evolve during stochastic gradient descent more accurately. These equations reveal when learning becomes unstable and explain why real training behaves differently than idealized continuous models. Their work helps predict learning progress and instability better than previous simpler models.
What this means in practice
- •For machine learning engineers: Improve tuning of learning rates and batch sizes by accurately predicting training stability and parameter dynamics during stochastic gradient descent.
- •For optimization algorithm developers: Design new algorithms that incorporate parameter covariance and curvature effects to enhance training robustness and convergence speed.
Authors
Christian Schmid, James M Murray
Abstract
Analytical theories of learning in deep networks often assume gradient flow, discarding the finite step size and batch noise of stochastic gradient descent (SGD). Starting from the discrete update, we derive continuous-time equations for the mean and covariance of the parameter distribution under a Gaussian closure for a general stochastic learning, to first order in the learning rate $η$. Applying these stochastic learning flow (SLF) equations to SGD, we show that the mean descends an effective loss regularized by the covariance and loss curvature, while the covariance, unlike in stochastic differential equation approximations, becomes unstable at curvature $2/η$. In SGD ensembles of linear and nonlinear networks, SLF predicts learning curves and mean trajectories more accurately than gradient flow and captures the covariance dynamics, failing where the parameter distribution becomes strongly non-Gaussian. In the full-batch limit, the edge of stability corresponds to a stationary point of the coupled sharpness-covariance dynamics, recovering self-stabilization and central flow, and batch-to-batch curvature fluctuations lower this threshold under SGD.