Efficiently exploring safely in reinforcement learning with new estimator
Computationally efficient safe exploration in reinforcement learning
Machine LearningRobotics
Summary
Reinforcement learning teaches computers to make choices, but trying new things can be risky in real life. The authors created a new method that helps the computer explore safely while learning, using a smart way to estimate outcomes that is faster and needs less computing power than previous methods. They tested this in a simple grid world and on data from Mars terrain, showing it works well without heavy calculations.
What this means in practice
- •For robotics engineers: Implement safe learning algorithms in robots navigating unknown or risky environments with reduced computation requirements.
- •For geospatial analysts: Use efficient safe exploration techniques to improve decision-making with spatial data such as planetary terrain observations.
Authors
Shreeram Murali, Shankar A. Deka, Dominik Baumann
Abstract
Reinforcement learning in real-life applications requires safety guarantees during exploration. Typical reinforcement learning algorithms do not provide such guarantees, and many modifications that do rely on Gaussian processes (GPs), which have a large computational cost. We propose a computationally lightweight algorithm based on the Nadaraya-Watson estimator that safely explores and optimizes constrained Markov decision processes (MDPs). Our algorithm, \textsc{CoLSafe-MDP}, uses an estimator that scales in constant-time with bounds on the estimates, a significant improvement from its GP-based counterparts that scale cubically with the number of data points. We then evaluate its performance in a grid-based environment and on observational Martian terrain data.