Neural actor critic improves continuous control with refined sampling
Neural Actor--Critic Learning with Minimax Critics and Reflected Langevin Sampling
Machine Learning
Summary
Learning how to make decisions in complex situations with many possible actions is hard because you need accurate guesses about both the value of actions and how to improve them. The authors propose a new method that combines a special kind of neural network estimator and a sampling strategy to better learn continuous action policies. They prove that their approach controls errors well and tested it in different simulated settings where it outperforms some existing methods. This work helps improve the reliability of learning policies that choose actions smoothly and effectively.
What this means in practice
- •For robotics engineers: Develop control policies that smoothly handle continuous movements using improved actor-critic learning with error guarantees.
- •For autonomous vehicle developers: Improve decision systems for vehicles operating in complex environments by applying refined policy-gradient and sampling techniques.
Authors
Kunqi Huang, Lican Kang, Hongrui Shu, Jincheng Ying
Abstract
Actor--critic methods rely on critic gradients to improve continuous-action policies, yet finite-sample guarantees must control both value and gradient errors and their propagation through the actor. We develop a neural actor-critic procedure that combines a regularized minimax Bellman estimator over sigmoid networks with a finite reflected Langevin sampler driven by the fitted critic gradient. Under uniform Bellman regularity and covered reset sampling, a positive high-order action-derivative penalty yields value and action-gradient risk bounds with classical nonparametric minimax exponents up to logarithmic factors. We propagate estimation, mixing, discretization, temperature, and policy-iteration errors to obtain return guarantees, with refinements for bounded multimodal potentials and strongly concave regimes. Empirical studies across controlled simulations, logged continuous-control data, and a simulated decision task show favorable performance relative to the considered baselines.