Reinforcement learning improves safe navigation in unknown environments

RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments

Robotics

Summary

Navigating safely through unknown places is tricky because robots must avoid collisions while figuring out where to go. The authors combine two techniques: reinforcement learning, which helps robots learn from experience, and a special planning method that ensures safety with high probability. Their approach trains models to predict sensor data and control actions, then plans moves that reduce risks while aiming for good outcomes. They tested this both in simulations and with a flying robot, showing improved safety and better navigation in complex settings.

What this means in practice

  • For robotics engineers: Enable robots to navigate unknown environments safely by combining learned sensor models with control methods that guarantee low collision risk.
  • For autonomous drone operators: Improve vision-based flight control in drones flying through unfamiliar areas by applying probabilistic safety assurances alongside reinforcement learning.

Authors

Adam Polevoy, Dillon Capalongo, Katherine Tang, Mark Gonzales, Marin Kobilarov, Joseph Moore

Abstract

In this paper, we present an approach for combining stochastic nonlinear model predictive control (SNMPC) and reinforcement learning (RL) to enable probabilistically-safe perception-based navigation in unknown environments. Our method first uses RL to train probabilistic actor-critic and sensor prediction models. We then leverage these probabilistic models in a sampling-based SNMPC framework known as Probably Approximately Correct (PAC)-NMPC, which uses hard constraints to enforce finite-time statistical guarantees on the probability of collision and value function improvement. By ensuring that our finite-horizon SNMPC policies decrease the value function in expectation, we can approach the long-horizon performance of the RL approach while satisfying probabilistic safety constraints. Through simulation experiments, we show that our approach can improve the safety of perception-based RL navigation policies and scale to high dimensional systems with large sensor input spaces and complex nonlinear dynamics. We also demonstrate our approach through hardware experiments, showing improved performance for vision-based navigation with an agile fixed-wing aerial vehicle in unknown environments.