Certifiable Safe Model-Based Reinforcement Learning with Control-Affine Dynamics Approximation
Robotics
Summary
The authors developed a method to help robots learn how to control themselves safely when the exact details of their movements are not fully known. They focus on a special type of robot model where the effect of control actions is clear and can be efficiently learned using a technique called Control-Affine Random Fourier Features. To ensure the robot stays safe while learning, they use a way to measure uncertainty in the model called adaptive conformal prediction combined with control barrier functions. Their approach was tested successfully on simulated robots like a cartpole and a quadrotor. This shows a promising way to balance learning efficiency with safety guarantees in robot control.
Authors
Hao Zhou, Yanze Zhang, Cameron Reid, Wenhao Luo
Abstract
Safe model-based reinforcement learning (RL) often bridges control-theoretic analysis and RL for robots to safely explore (partially) unknown system dynamics while deriving control actions for task efficiency. The control performance and safety assurance typically rely on prior knowledge of partially modeled nominal system dynamics and the data-driven models that compensate for residual model uncertainties. However, existing methods often overlook the structure of residual model uncertainties (e.g., components affine in control), which could lead to overly conservative robot behaviors or invalid safety guarantees under the safe learning-based controllers. This paper proposes a safe reinforcement learning framework that learns control-affine dynamics with a certifiable data-driven safe policy using control barrier functions (CBF). Specifically, we first use Control-Affine Random Fourier Features (ARFF) to model robot dynamics in a control-affine form, which offers computational efficiency that scales with dataset size and reduces potential model bias for model-based reinforcement learning. Then, a model-free, efficient uncertainty quantification method using adaptive conformal prediction (ACP) is applied to quantify the uncertainty in the safety constraint arising from the learned control-affine dynamics. This allows for data-driven safety assurance amenable to principled and efficient controller synthesis with CBF. Simulation results on the cartpole and the 3D quadrotor platforms demonstrate the effectiveness of the proposed framework.