Geometric moment contraction improves understanding of stochastic nesterov method
Geometric Moment Contraction for Stochastic Nesterov Acceleration
Machine Learning
Summary
Optimizing complex systems often involves random or noisy data, which can make it hard to know if an algorithm will reliably improve over time. The authors study a well-known acceleration method called stochastic Nesterov and establish precise conditions under which it consistently converges, even with noisy information and varying mathematical properties. They develop theoretical tools to measure this contraction behavior and identify how different parameters affect stability and speed. This work helps clarify when and how the method can be trusted to perform well under uncertainty.
What this means in practice
- •For machine learning engineers: Design and tune accelerated stochastic optimization algorithms with guaranteed convergence under noisy gradients.
- •For control systems designers: Analyze stability and robustness of accelerated feedback schemes in uncertain environments using moment contraction criteria.
A theory result. No direct application yet.
Authors
Wei Biao Wu
Abstract
We study geometric moment contraction (GMC) of the constant-parameter stochastic Nesterov recursion \[ Y_k=Θ_k+β(Θ_k-Θ_{k-1}),\qquad Θ_{k+1}=Y_k-γG(Y_k,X_{k+1}). \] Under mean strong monotonicity and stochastic $L^p$ Lipschitz continuity, an explicit Perron comparison proves synchronous $L^p$ contraction when $βγL_p<(1-β)(1-q_{γ,p})$. This direct criterion includes infinite-variance gradients for $1<p<2$, but its small-step regime requires $β<μ/(μ+L_p)$. A complementary power-Lyapunov argument establishes a positive, generally much smaller, step-size interval for every fixed $β<1$ and every $p>1$, using only a finite $p$th gradient moment. At $p=2$, a simpler explicit certificate gives \[ 0<γ<\frac{2μ(1-β)^2}{L_2^2(1-β+2β^2)}. \] Its quadratic high-momentum scaling is a limitation of the chosen metric, not a sharp stability boundary. We quantify this loss, provide a general mean-only quadratic $S$-procedure, and exploit endpoint Lyapunov inequalities under stronger samplewise sector information. Verified endpoint certificates can be orders of magnitude less conservative than the explicit metric.