Gaussian behavior emerges in SGD despite infinite stationary moments

Gaussian Limits for SGD Without Stationary Moments

Machine Learning

Summary

Stochastic gradient descent (SGD) is a common method to train models, but understanding its long-term error distribution can be tricky when errors do not settle down nicely. The authors show that even when usual measures of error size become infinite due to dependencies in data, the overall SGD error still behaves like a Gaussian (bell curve) under certain conditions. They develop a mathematical theory explaining when and how this Gaussian behavior appears, helping us better approximate and understand the error distributions beyond standard assumptions. This improved understanding is supported by experiments checking how well these approximations work.

What this means in practice

  • For machine learning engineers: Accurately characterize the error distribution of SGD in complex settings with dependent data to improve model diagnostics and validation.
  • For statistical inference teams: Use refined Gaussian limits to enhance confidence interval calibration when model errors have unusual dependence structures.

A theory result. No direct application yet.

Authors

Xiaoli Li, Wei Biao Wu

Abstract

Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with $o(a)$ error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.
View PDFOpen arXiv