Actor critic algorithm maintains reliable learning with high probability
A Concentration Bound for Two-Timescale Actor-Critic Algorithm
Machine Learning
Summary
This paper addresses how a type of machine learning algorithm called actor-critic, which learns by updating two parts at different speeds, behaves over time. The authors developed a mathematical guarantee that with high probability, the algorithm’s 'actor' part stays close to the best solution after enough updates. They also confirmed with experiments that the error indeed gets smaller as learning progresses. This helps increase trust in using actor-critic methods for long-term decision tasks.
What this means in practice
- •For reinforcement learning engineers: Assess stability and error bounds when deploying two-timescale actor-critic algorithms in continuous control tasks.
- •For robotics developers: Improve confidence in long-term policy learning for robots that use actor-critic methods.
Authors
Prashansa Panda, Shalabh Bhatnagar
Abstract
Significant research effort has been directed in recent years towards establishing both asymptotic and non-asymptotic convergence guarantees for two-timescale actor--critic algorithms, where the actor recursion is run on a slower timescale than the critic recursion. This work derives a uniform all-time concentration bound for the actor--critic algorithm with function approximation in the long-run average-reward setting. This bound helps us analyze the behavior of the actor parameter with high probability. We show that, after some finite time, the actor parameter enters a safe region and remains within it thereafter with high probability. Specifically, with probability at least $1-ε_1-ε_2$, the actor error $\Vert θ_k-θ^{*}\Vert$ is $O\left(\frac{n_0^{3/4}}{k}\frac{1}{\sqrt{ε_2}}+\left(\frac{1}{n_0}\right)^{1/4}\log^{1/4}\left(\frac{1}{ε_1}\right)+\left(\frac{1}{n_0}\right)^{1/4}\right)$ for all $k\geq n_0$ and sufficiently large $n_0$. We also present experimental results demonstrating that the aforementioned actor error diminishes with the number of actor-parameter updates.