Rvi q-learning achieves fast convergence for finite mdp policies
$O(1/n)$ Convergence Rate for RVI $Q$-Learning
Machine Learning
Summary
Finding the best way to make decisions in uncertain situations can be tricky and slow. This paper studies a mathematical method called RVI Q-learning that helps computers quickly learn good decision strategies. The authors prove that this method improves its guesses at a steady and fast rate, meaning better decisions come sooner. Their work also shows how many steps are needed to get a policy that is nearly the best. This helps make algorithms for decision making more reliable and efficient.
What this means in practice
- •For reinforcement learning engineers: Use the convergence rate to design faster training schedules for synchronous RVI Q-learning in finite MDP problems.
- •For robotics software teams: Implement RVI Q-learning with guaranteed performance bounds to improve decision-making speed in robot navigation tasks with finite state spaces.
Authors
Avik Kar, Siddharth Chandak, Rahul Singh, Shalabh Bhatnagar
Abstract
We establish finite-time convergence guarantee for synchronous relative value iteration (RVI) $Q$-learning for finite weakly communicating Markov decision processes (MDPs). For stochastic Krasnosel'skiĭ--Mann iterations driven by a non-expansive, linearly regular map on a finite-dimensional normed space, we establish last-iterate mean-squared residual bounds under martingale-difference noise whose conditional second moment grows at most linearly with the squared residual. Suitable polynomial stepsizes of order $n^{-a}$ yield an $O(n^{-a})$ rate for $0<a\leq1$. We verify linear regularity and the noise condition for the centered Bellman operator for finite weakly communicating MDPs. Applying the general result gives an $O(n^{-1})$ bound on the mean-squared Bellman span residual of RVI $Q$-learning iterates when a suitable stepsize sequence is used. The positive gap between optimal and suboptimal stationary deterministic policies then yields an $O(n^{-1})$ bound on the expected suboptimality of the greedy policy. Consequently, $O(ε^{-1})$ synchronous iterations suffice for an $ε$-optimal policy in expectation. These guarantees are instance-dependent, with constants that depend on the hitting-time parameters and the suboptimality gap of the second-best policy.