Clipped SGD achieves faster and more accurate convergence bounds
Precise Convergence Speed of Clipped SGD
Machine Learning
Summary
When training machine learning models, a common technique is to adjust model settings gradually by following gradients. Sometimes these gradients are 'clipped' to avoid very large steps. This paper by the authors explains more precisely how this clipped gradient method converges towards a solution, improving on previous analyses. They provide clearer mathematical constants and conditions for stable training, and also improve the way we measure progress over many steps. This helps better understand and tune training to be both faster and more reliable.
What this means in practice
- •For machine learning engineers: Set more precise learning rates for clipped gradient training to improve stability and speed of machine learning model optimization.
- •For optimization algorithm designers: Use the refined convergence and bias controls in developing and analyzing new stochastic optimization methods involving clipping.
A theory result. No direct application yet.
Authors
David A. R. Robin
Abstract
We present a tightened convergence analysis of clipped gradient descent on $(L_0, L_1)$-smooth functions, with quantitative constants. Building on the ideas of Koloskova et al (2023), we refactor several case disjunctions to reveal the central role of a control of the bias derived from fundamental properties of $\ell_2$-projection, simplifying proofs. We also extend the domain of validity from $η\leq 1 / (9 β)$ to $η< 1 /β$ where $β= L_0 + c L_1$ for clipping constant $c$, which matches the more traditional analysis of smooth functions. We strengthen the convergence criterion from $\left( \min_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ to $\left( \frac{1}{T} \sum_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ with matching speed, and lower the final achievable loss from $\mathcal{O}(\min(σ^2/c, σ))$ to the more precise $6 \min(σ^2 /c, 3 σ)$.