Robust Peak-cost Constrained Reinforcement Learning
Machine Learning
Summary
The authors study a type of reinforcement learning where the goal is to get high rewards without ever having a big safety violation during a task, which is important in critical situations. They show that the usual math tricks used for balancing rewards and costs don't work perfectly here because big cost limits behave differently than average costs. To handle uncertainty between simulations and the real world, they propose a new method that estimates values more robustly. Their approach guarantees near-optimal rewards while keeping safety violations very small, and their experiments show it works well even when conditions change.
Authors
Shilpa Mukhopadhyay, Sourav Ganguly, Santosh Mohan Rajkumar, Honghao Wei, Debdipta Goswami, Arnob Ghosh
Abstract
We study robust peak-cost constrained reinforcement learning (RP-CRL), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory. This setting is motivated by safety-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost. Existing reachability-constrained RL methods adopt Lagrangian-based approaches, yet the underlying duality properties of peak-cost constrained MDPs remain unclear. We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap. We further consider a robust formulation to address simulator-to-real-world mismatch in the transition dynamics. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at most epsilon. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance.