Meta reinforcement learning improved with safety awareness in new method
Safe Meta-Reinforcement Learning via Information Space Reachability
Machine Learning
Summary
Reinforcement learning teaches computers to learn tasks by trying them and getting rewards or penalties. However, it can be risky when the computer tries new, unknown tasks because it might make unsafe decisions. The authors propose a new way to help these learning computers understand safety better by including both what they see and what they believe about the task. This approach helps them avoid dangerous situations while learning quickly. Tests show this method helps computers stay safe while adapting to new challenges.
What this means in practice
- •For robotics engineers: Develop robots that adapt safely to new tasks by avoiding unsafe actions during learning and operation.
- •For autonomous vehicle developers: Improve safety filters in self-driving cars to maintain safe behavior when encountering unfamiliar driving scenarios.
Authors
Zeyang Li, Sunbochen Tang, Navid Azizan
Abstract
Meta-reinforcement learning (meta-RL) enables agents to adapt to unseen tasks with limited experience. Despite its promise, the application of meta-RL in real-world tasks is hindered by safety requirements, which have been underexplored in prior work. In this paper, we propose a safe meta-RL framework that explicitly accounts for safety during adaptation. Our key insight is to reason about safety in the information space, which captures both the physical state and the agent's belief over the underlying task. Within this space, we introduce a safety value function that measures the probability of the agent avoiding unsafe regions indefinitely. We show that this function satisfies a self-consistency condition and a Bellman equation, which make it learnable via meta-RL. Based on this formulation, we develop a safe meta-RL algorithm that learns the safety value function and leverages it for safety filtering and constrained policy optimization. Experiments on meta-RL benchmarks demonstrate the effectiveness of the proposed method.