Model based learning improves robot control with planner guided value updates
CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning
Robotics
Summary
Robots need to learn how to make good decisions efficiently, and one way is to build a model of their environment. The authors found that combining this model with a type of planning helps robots choose better actions. They designed a new method called CAST that improves how the robot learns the value of different states by mixing information from planned actions and current policies. This approach was tested in various simulated tasks and even transferred successfully to a real robot doing a dynamic move. Their method aims to balance using strong planned behavior while still learning from the robot's own experiences.
model-based reinforcement learningonline planningvalue functionstate-value criticaction-value criticpolicyplanner-guided behaviorsample efficiencyrobot control
Authors
Pietro Noah Crestaz, Mohamed Yassine Kabouri, Nicolas Mansard, Andrea Del Prete
Abstract
Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.