Model based reinforcement learning improves robot control with planner guided value updates
CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning
Robotics
Summary
Controlling robots efficiently often requires learning not just from experience but by imagining future actions. The authors introduce CAST, a method that helps machines learn better by mixing plans made using a model with current learned behavior. CAST teaches the robot to value states based on a blend of real planner decisions and imagined choices, making the robot act more skillfully. They tested CAST in simulated control tasks and a real robot doing a physical handstand, showing it learns better movement strategies.
What this means in practice
- •For robotics engineers: Improve sample efficient robotic control by combining model based planning with policy learning for more reliable dynamic movements.
- •For robotics hardware developers: Deploy learned locomotion skills tested in simulation directly onto physical quadruped robots for dynamic tasks like handstands.
Authors
Pietro Noah Crestaz, Mohamed Yassine Kabouri, Nicolas Mansard, Andrea Del Prete
Abstract
Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.