Safe multi-objective reinforcement learning with zero constraint violations
Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence
Machine LearningArtificial Intelligence
Summary
Balancing multiple goals safely is a big challenge when teaching computers to make decisions over time. The authors developed new methods that guarantee no safety rules are broken while the computer learns and explores. Their methods also find good trade-offs between goals, even when some preferences aren’t known at the start. This helps ensure safe and effective learning in complex environments where making mistakes can be costly or dangerous.
What this means in practice
- •For autonomous vehicle engineers: Develop vehicle control systems that balance speed, fuel efficiency, and safety without violating safety limits during training.$Commercial implications: Enables production-ready self-driving car systems that rigorously guarantee safety during development and deployment.
- •For robotics system developers: Create robot behaviors that safely manage multiple objectives like energy use and task completion without unsafe actions during learning.
Authors
Chenxi Li, Yao Shu, Siyang Gao, Shuang Qiu, Guangxin Jiang
Abstract
Many applications of multi-objective reinforcement learning (MORL) require policies that balance conflicting objectives while satisfying safety constraints. We study safe MORL with zero constraint violation throughout learning, covering both cumulative and stepwise constraints. A key challenge is to explore sufficiently under unknown dynamics and constraint costs without violating the constraints. For a given preference over objectives, we develop an algorithm to learn a safe Pareto optimal policy by combining optimistic reward estimates with pessimistic cost bounds. With high probability, every policy executed over $T$ episodes incurs zero constraint violation. The algorithm returns a safe Markov policy induced by the averaged occupancy measure and achieves an $\widetilde{O}(1/\sqrt{T})$ suboptimality gap under smooth Tchebycheff scalarization. We further develop a preference-free algorithm that first explores safely without a prespecified preference and then computes a safe policy for any given preference without further interaction. Over $T$ exploration episodes and $K$ planning iterations, it retains the same safety guarantee and achieves an $\widetilde{O}(1/\sqrt{T}+1/\sqrt{K})$ suboptimality gap. Both algorithms require knowledge of a policy that strictly satisfies the constraints and its costs. We then relax these requirements, incurring at most constant cumulative violation. To our knowledge, we provide the first sample-efficient MORL algorithms that guarantee safety for every executed policy.