Robust reinforcement learning policies improve long term safety and performance
Robust Policy Optimization via Adversarial Importance Sampling
Machine Learning
Summary
Reinforcement learning systems can struggle when the data or inputs they receive are slightly changed in tricky ways. The authors introduce a new method called Adversarial Importance Sampling (Advis) that helps train these systems to be stronger against such changes without extra trial runs or complex additions. They also made a tool called advrl that makes testing different safety techniques easier. Their research shows that testing with more varied tricky inputs is important to truly check how tough these systems are. They tested their method in simulated control tasks and found it worked better than older methods.
What this means in practice
- •For autonomous vehicle developers: Improve the reliability of control policies against unexpected sensor changes without extra costly vehicle testing.
- •For robotics engineers: Train robots to maintain stable behavior under variable or disturbed inputs using fewer real-world experiments.
Authors
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Abstract
Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: https://github.com/AmineAndam04/advrl