Gradient clipping improves optimization under heavy tailed noise
Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails
Machine Learning
Summary
When computers learn from data, random noise can make the learning unpredictable, especially if this noise has extreme values that happen often. The authors studied a method called gradient clipping that trims very large updates during learning. They show that this technique works well even when the noise is unusually heavy-tailed and depends on the learning steps themselves. Their results give guarantees about how quickly the learning process improves, and they confirm these findings with experiments using a type of extreme noise distribution.
What this means in practice
- •For machine learning engineers: Improve training reliability of models by controlling extreme gradient updates under heavy-tailed, data-dependent noise.
- •For data scientists in finance: Analyze optimization methods robust to heavy-tailed, scale-dependent noise common in financial data modeling.
Authors
Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov, Alexander Gasnikov, Margarita Preobrazhenskaia, Igor Preobrazhensky, Eduard Gorbunov
Abstract
Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Existing analyses typically assume a bounded central moment of order $α\in(1,2]$, which excludes distributions with tails of order $u^{-α}$ at the critical exponent. In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm. We derive high-probability convergence guarantees for clipped SGD (Clip-SGD) and its accelerated variant Clip-SSTM under a weak-$L^α$ tail condition, which is strictly weaker than the corresponding moment bound. For $1<α<2$, the clipping bounds have the same powers of the clipping level as in the moment case, and both methods attain the additive stochastic rate $K^{-(α-1)/α}$. The gradient-dependent noise contributes a faster-decaying term of order $K^{-2(α-1)/α}$, up to logarithmic factors. At $α=2$, we characterize the additional logarithmic factor on the Lorentz scale and establish its necessity for the additive error at fixed confidence. We also analyze Clip-SGD with prescribed clipping levels and derive finite-horizon guarantees for reaching a neighborhood determined by the clipping bias. For additive noise, matching lower bounds show that this neighborhood is unavoidable, up to constants and the natural error cap. Numerical experiments with Pareto noise illustrate the theoretical results.