AI summaryⓘ
The authors study a type of algorithm called stochastic approximation (SA) that is used in areas like reinforcement learning, where randomness and changing noise levels make analysis tricky. They focus on situations where the noise scales with the current estimate and the estimates can grow without bounds, which is common in practice. Instead of using complicated smoothing techniques from earlier work, the authors present a simpler and unified way to analyze both average performance and how likely large deviations are. Their method uses new tools like an averaged noise sequence and probabilistic induction, leading to the first strong concentration results with sub-Gaussian tails for SA under these noise conditions. They also discuss how their approach might extend to other related algorithms and noise types.
Stochastic approximationMultiplicative noiseConcentration boundsMean-square boundsContractive mappingsLyapunov driftAzuma-Hoeffding inequalityReinforcement learningGeneralized Moreau envelopeProbabilistic induction
Abstract
We establish mean-square and concentration bounds for stochastic approximation (SA) with arbitrary norm contractive mappings, under a multiplicative noise model where the noise may scale affinely with the norm of the iterates, and the iterates are potentially unbounded. These settings arise in reinforcement learning, where operators are often contractive in the $\ell_\infty$ norm and the noise scales with the iterates. To address the arbitrary norm, earlier works replace the non-smooth squared norm with a smooth Lyapunov function constructed via the generalized Moreau envelope. For concentration analysis, these works handle multiplicative noise and unbounded iterates through a multi-stage bootstrapping argument that starts from a time-varying worst-case bound and iteratively refines it. We instead present a unified and elementary analysis that yields both bounds. Using an averaged noise sequence and corresponding auxiliary iterates, we obtain a one-step Lyapunov drift inequality for the normed error directly, without smoothing the norm or constructing an envelope. For the mean-square bound, we combine this drift inequality with an induction argument showing that the iterates remain bounded in expectation. For the concentration bound, we develop a probabilistic induction over a sequence of "good" events on which the iterates are controlled, allowing the standard Azuma-Hoeffding bound to be applied. Our approach yields the first sub-Gaussian tailed maximal (all-time) concentration bound for SA under multiplicative noise, by allowing the stepsize to depend logarithmically on the confidence level. Beyond the specific setting considered here, we discuss the generalizability of these proof techniques to other noise models and iterative algorithms.