The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System
2026-07-06 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explore how gradient descent, a method used to train deep learning models, behaves as a discrete process rather than a smooth one. They use simplified models that keep key deep learning features to show that certain training behaviors, like oscillations and stability edges, come from the training process itself and not from failure. Their work reveals that the learning rate affects which solutions the training finds by influencing the dynamics and resulting representations. These findings hold even with factors like data complexity, nonlinearities, and randomness. Overall, the authors show that the learning rate shapes how models learn, beyond just ensuring stable updates.
gradient descentedge of stabilitydiscrete dynamical systemdeep linear networkslearning ratebifurcationspectral edgesrepresentation learningoscillationsfactorization
Authors
Thomas Hofmann
Abstract
Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them. Edge-of-stability behavior, sharpness oscillations, catapult phases, balancing, and movement toward flatter representations are effects of the training map itself, and are poorly captured by the small-step gradient-flow limit. This paper studies fixed-step gradient descent as a discrete dynamical system in a hierarchy of exactly solvable models retaining basic structures of deep learning: depth, factorization, width, data coupling, activation, and stochasticity. The starting point is the balanced scalar reduction of a deep linear chain, giving a quartic loss and a cubic gradient map whose post-edge behavior is explicit. Under the natural large-depth scaling, this dynamics converges to a universal Ricker-type map. The edge of stability is therefore not a breakdown of optimization, but the first bifurcation of the training map. Embedding the scalar dynamics back into factored models turns these regimes into learning phenomena. Finite steps break conservation laws of gradient flow and contract factorization imbalance; residual oscillations move parameters toward flatter, more balanced representations. Wider linear networks produce a ladder of spectral edges, so the optimal learning rate can lie beyond the first edge. Data coupling, nonlinear activations, and stochastic targets preserve the same organizing principle: finite-step oscillations drive alignment, balancing, and representation selection. Thus the learning rate is not merely a numerical stability parameter. It is a structural parameter of the training dynamics, determining its attractors and shaping the representations gradient descent selects.