Stochastic gradient descent converges in deep linear neural networks
Convergence of stochastic gradient descent for learning linear neural networks
Machine Learning
Summary
Stochastic gradient descent (SGD) is widely used to train deep learning models, but it’s hard to understand exactly how and why it works, especially with complicated models. The authors studied a simpler type of neural network called deep linear networks, which are easier to analyze but still share some real challenges. They proved that when using SGD with certain conditions, the process will reliably get closer to stable points where learning doesn’t improve anymore. This work helps bridge the gap between theoretical math and practical training of neural networks.
What this means in practice
- •For machine learning engineers: Design training routines for deep linear models with guarantees that stochastic gradient descent will converge reliably under mild conditions.
- •For optimization algorithm developers: Develop improved stochastic optimization algorithms informed by theoretical convergence results in non-convex linear network settings.
A theory result. No direct application yet.
Authors
Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut, Ulrich Terstiege
Abstract
Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.