Adam optimizer loss spikes linked to momentum and loss shape
Beyond Quadratic Loss: The Stability Phase Diagram of Adam
Machine Learning
Summary
Training deep learning models sometimes experiences sudden jumps in error called loss spikes, which slow down learning. The authors studied the Adam optimizer and found that these spikes depend on how two kinds of momentum are balanced during training. They discovered a simple mathematical relationship that separates stable training from unstable, spiky training. They also showed that the shape of the error landscape beyond simple curvature explains these spikes, especially for common classification losses.
What this means in practice
- •For machine learning engineers: Tune Adam optimizer hyperparameters beta1 and beta2 more effectively to avoid sudden training instabilities.
- •For ai model trainers: Diagnose and mitigate loss spikes in classification tasks by understanding loss landscape shape beyond quadratic approximations.
Authors
Gaoxiang Tang, Huanran Chen, Ziming Liu
Abstract
Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the $(β_1,β_2)$ plane. Across a range of model--task settings, an approximately linear boundary, $1-β_2=C(1-β_1)$, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss $L(x)\propto|x|^n$ recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent $n$. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.