Clipped SGD achieves faster and more accurate convergence bounds

Precise Convergence Speed of Clipped SGD

Machine Learning

Summary

When training machine learning models, a common technique is to adjust model settings gradually by following gradients. Sometimes these gradients are 'clipped' to avoid very large steps. This paper by the authors explains more precisely how this clipped gradient method converges towards a solution, improving on previous analyses. They provide clearer mathematical constants and conditions for stable training, and also improve the way we measure progress over many steps. This helps better understand and tune training to be both faster and more reliable.

What this means in practice

A theory result. No direct application yet.

Authors

David A. R. Robin

Abstract

We present a tightened convergence analysis of clipped gradient descent on $(L_0, L_1)$-smooth functions, with quantitative constants. Building on the ideas of Koloskova et al (2023), we refactor several case disjunctions to reveal the central role of a control of the bias derived from fundamental properties of $\ell_2$-projection, simplifying proofs. We also extend the domain of validity from $η\leq 1 / (9 β)$ to $η< 1 /β$ where $β= L_0 + c L_1$ for clipping constant $c$, which matches the more traditional analysis of smooth functions. We strengthen the convergence criterion from $\left( \min_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ to $\left( \frac{1}{T} \sum_{t < T} \mathbb{E}[\lVert \nabla f(x_t) \rVert_2] \right)$ with matching speed, and lower the final achievable loss from $\mathcal{O}(\min(σ^2/c, σ))$ to the more precise $6 \min(σ^2 /c, 3 σ)$.