Safe reinforcement learning needs better safety measurements
Evaluation Metrics for Safe Reinforcement Learning
Artificial IntelligenceMachine Learning
Summary
Safe reinforcement learning tries to teach machines to make good choices without causing harm. The usual way to check if these machines are safe looks only at average safety, which misses important details like how often and how badly safety rules are broken. The authors introduce new ways to measure safety more thoroughly and a system to compare different safety techniques. They tested these ideas on multiple tasks and found that using several measurements together gives a clearer picture of safety. They also created an open-source tool to help others check safety more reliably in future work.
What this means in practice
- •For autonomous vehicle teams: Evaluate autonomous driving systems to better predict and reduce safety violations during development and after deployment.
- •For robotics engineers: Benchmark robot decision-making algorithms on varied safety criteria to ensure consistent safe behavior under different tasks and constraints.
Authors
Lindsay Spoor, Aske Plaat, Thomas Moerland
Abstract
Safe reinforcement learning (RL) is commonly formalized as a Constrained Markov Decision Process (CMDP), in which an agent maximizes expected reward while keeping its expected cumulative cost below a specified safety bound. Existing safe RL benchmarks predominantly report whether an algorithm is safe on average, following this expectation-based guarantee. We argue that this convention is insufficient to reliably characterize an algorithm's true safety: it fails to capture how often and how severely the safety bound is violated, whether this holds consistently across tasks and safety bounds, and whether training-time behavior is representative of behavior of the final converged policy. Therefore, we introduce (i) evaluation metrics for safe RL that address each of these concerns and in addition allow for aggregation across tasks and safety bounds. We furthermore define (ii) a safety tier system to systematically categorize and compare algorithms in terms of safety and reliability at both training and for a final policy. Using this framework, we provide (iii) an empirical safety evaluation across multiple safety navigation tasks. Our results show that aggregate metrics, distributional reporting, and task- and safety bound-specific results each reveal information the other metrics cannot. We therefore recommend reporting all three jointly, rather than compressing this information into a single value, as is common practice. We provide SafeRLEval, an open-source evaluation suite to support the reliable characterization of safety in future safe RL research.