Reinforcement learning methods make robots safer in real tasks

Towards Probabilistically Safe Reinforcement Learning: Theory and Deployment on Humanoid and Aerial Robots

Robotics

Summary

Robots can learn to do complex tasks with reinforcement learning, but making sure they stay safe during learning is hard. The authors improved a method that helps robots learn while keeping safety rules, even when the learning signals are noisy or biased. Their approach works with common algorithms and can handle tasks like avoiding collisions with some allowed risk. They tested it on a humanoid robot and a flying drone, showing safe behaviors right after simulating.

What this means in practice

  • For robotics engineers: Deploy whole-body control policies on humanoid robots with guaranteed safety boundaries during learning and use in real environments.
  • For drone navigation developers: Enable collision-free waypoint navigation for drones using reinforcement learning methods that respect probabilistic safety constraints.

Authors

Pol Mestres, Varun Gabbita, Jason H. K. Siu, Sergio A. Esteban, Hampus Serneke, Joel W. Burdick, Aaron D. Ames

Abstract

Reinforcement learning (RL) enables robots to acquire complex skills, but standard RL formulations do not guarantee that learned policies satisfy safety constraints, which limits their use in safety-critical settings. Existing safe RL methods with provable guarantees typically support only restricted classes of constraints and require unbiased estimators of the objective and constraint functions, which prevents their direct integration with state-of-the-art algorithms such as proximal policy optimization (PPO). In this paper, we extend the recently introduced robust safe gradient flow-based reinforcement learning (RSGF-RL) framework to overcome these limitations. Our formulation accommodates a general class of trajectory-level safety constraints, including chance constraints such as collision avoidance with a prescribed probability, and our analysis allows for biased value and gradient estimators. We show that, with high probability, the constraint violation throughout training is bounded by the estimator bias plus a term that vanishes as the number of rollouts grows, and decays to this level if the initial policy is unsafe. We also establish the convergence properties of the iterates to approximate KKT points of the original safe RL problem. Finally, we show that RSGF-RL implemented with PPO-style estimates enables zero-shot sim-to-real deployment of whole-body control policies on a Unitree G1 humanoid and collision-free waypoint navigation on a Crazyflie drone.