Framework certifies reinforcement learning policies against sparse action attacks
FoSeRL: Formal Sequential Robustness Certification for Reinforcement Learning Policies
Machine Learning
Summary
Decision-making programs called reinforcement learning policies can fail if their actions are changed, even just a little. The authors introduce FoSeRL, a technique that checks whether a policy will still perform well when some actions are interfered with occasionally. Their method does not change the policy itself and works by comparing runs under the same conditions with and without attacks. They tested FoSeRL on several control tasks and found it gives reliable guarantees about performance loss and shows differences in robustness between policies that otherwise seem similar.
What this means in practice
- •For autonomous vehicle engineers: Ensure self-driving car controllers maintain safety and performance under occasional action disturbances by formally certifying worst-case return losses.
- •For robotics control teams: Evaluate and guarantee the robustness of robot control policies in uncertain and partially disrupted environments without retraining or modifying the controller.
Authors
Sara Taheri, Deep Kumar Ganguly, Jan Křetínský, Majid Zamani
Abstract
Even a few action perturbations can substantially degrade the performance of a deployed decision policy. Certifying the resulting return loss is challenging in stochastic environments, where returns vary even without an attack. We introduce FoSeRL, a framework for certifying deployed RL policies against precommitted, temporally sparse action attacks. The deployed policy is unchanged, with no smoothing or retraining. Certification requires a resettable simulator supporting shared randomness and independent one-step successor queries, but no analytical dynamics model. FoSeRL certifies that an attacked episode loses no more than a prescribed amount of return relative to the same episode unattacked, with at least a target probability and at a user-specified confidence level. Both runs share the initial state and randomness, so the measured loss reflects the attack, not the episode; carrying the running return gap as a state coordinate makes it the terminal value, reducing trajectory-level certification to terminal safety. Time-dependent barrier conditions on the augmented state bound the terminal failure probability: satisfied exactly, they certify every admissible precommitted attack; learned from sampled trajectories and verified on held-out data, they certify the same guarantee under a specified attack-episode setting. Across six stochastic continuous-control environments and three RL policy families (TD3, SAC, and PPO), FoSeRL certifies non-trivial cardinality--magnitude robustness frontiers, achieves substantially larger certified budgets than policy smoothing, and reveals marked robustness differences among policies with comparable nominal performance.