Importance-corrected momentum method improves actor-critic learning analysis
A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
Machine LearningArtificial Intelligence
Summary
Learning good strategies in environments that change over time is a hard problem. This paper studies a new combination of techniques called importance-corrected momentum and ridge-regularized critics to improve how algorithms learn from sequences of data. The authors provide new mathematical guarantees on how fast these methods converge when learning from continuous experience. They also compare different ways to adjust learning steps and show when certain adjustments lead to better outcomes.
What this means in practice
- •For reinforcement learning engineers: Design actor-critic algorithms that ensure faster and more stable policy updates from continuous streams of experience.
- •For control system developers: Improve control policies by applying corrected momentum methods that require fewer tuning parameters and handle temporal data dependencies.
A theory result. No direct application yet.
Authors
Yanjie Dong, Zhou Su, Wei Wang, Gang Wang
Abstract
We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of $T$ consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order $\frac{1}{T}$. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over $K$ updates is bounded by ${\cal O}(K^{-1/3}+\frac{1}{T})$ for normalized steps and by ${\cal O}(K^{-1/4}+\frac{1}{T})$ for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor $\min\{m,n\}^{1/2}$ associated with an $m\times n$ actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.