Safe reinforcement learning certification improves deployment accuracy

DEPLOY-RL: Active Boundary Discovery and Conservative Certification for Deployable Reinforcement Learning in Safety-Critical Continuous Processes

Machine Learning

Summary

Safety-critical processes rarely use reinforcement learning (RL) controllers because it's unclear when an RL policy can be trusted as safe. The authors present a method called DEPLOY-RL that focuses on finding exactly where RL policies might fail instead of checking all possible situations equally. This targeted approach helps efficiently certify policies with statistical guarantees about safety, reducing wrong approvals and enabling safer deployment decisions. They show this works well in industrial simulations, cutting false safety approvals by about half while still allowing most policies to be used.

What this means in practice

Authors

Yeojin Jang, Minu Baek, Gihun Gil, Minsung Jung, Beomdo Park, Woohyeon Kwon, Hyeonseok Jang, Junseong Park, Neha Sengar, Andres Saurez, Sangkeum Lee

Abstract

Reinforcement learning (RL) policies often outperform classical controllers in simulation, yet rarely reach production in safety-critical processes. The barrier is that there is no principled way to answer "Is this policy safe to deploy?" with statistical guarantees. We introduce DEPLOY-RL, a post-training certification framework built on one key insight: deployment certification requires discovering where failures occur (boundary discovery), not measuring how much everywhere (uniform reconstruction). Our contributions: (1) a contract-coupled acquisition function that concentrates sampling on certification-critical boundaries, achieving approximately 2x sample efficiency with a semi-empirical ambiguity reduction bound (domain-calibrated convergence guarantee); (2) conformal risk control providing finite-sample false-go guarantees (<= alpha) under explicit deployment contracts; (3) a three-way decision framework (Deploy/No-Deploy/Abstain) with fail-safe PID/MPC fallback. In simulations on papermaking (industrial digital twin) and Tennessee Eastman (public benchmark), DEPLOY-RL achieves 4.4% false-go rate (vs. 8.6% for the best baseline) while retaining 88.6% policy coverage, the only method achieving <5% false-go with >85% coverage among 14 baselines under our evaluation protocol.
View PDFOpen arXiv