Papers for

stochastic optimization practitioners

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Stochastic approximation steady-state limits with markov noise and non-smooth functions

Steady-State Convergence of Stochastic Approximation

Abstract: For constant-stepsize stochastic approximation (SA), the iterates converge in distribution to a stationary law that depends on the stepsize $α.$ Steady-state convergence (SSC) concerns the limit of the scaled stationary distribution as $α\downarrow 0.$ Existing SSC theory requires i.i.d. or additive noise and global differentiability of the mean operator, and yields suboptimal rates. We develop a unified SSC theory for constant-stepsize contractive SA driven by Markovian, multiplicative noise, covering both locally differentiable and locally nondifferentiable mean operators. A key methodological contribution is a multi-step universality framework that progressively reduces the original stochastic recursion to tractable auxiliary dynamics while preserving its steady-state limit. Under local quadratic linearization at the fixed point, we obtain a Gaussian approximation of the scaled steady state at the optimal rate $O(\sqrtα)$ in Wasserstein-2 distance, which further gives finite-time Gaussian approximations for the raw iterates. In the locally nondifferentiable regime, we establish a general SSC result and show that the leading-order asymptotic bias can be of order $\sqrtα$, in contrast to the $α$-order bias in the smooth regime. We apply the theory to Markovian linear SA and asynchronous Q-learning, neither of which is covered by prior results. We further propose a bias-reduction scheme for Q-learning that requires no knowledge of the local smoothness regime, validated by numerical experiments.

Mon 14 SeptMachine Learning
The gist
Stochastic approximation is a method used to find solutions iteratively under randomness, but understanding their long-term behavior can be tricky. The authors study how these methods behave when the randomness is more complex and when the function being approximated is not perfectly smooth. They provide a unified way to describe the steady behavior and show that the nature of the function affects the accuracy of approximations. Their work covers new cases like certain learning algorithms and includes a way to reduce errors in those cases.
Open 2609.14922v1