New method improves robustness of deep reinforcement learning policies
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Machine Learning
Summary
Deep reinforcement learning can struggle when faced with unexpected or tricky inputs that try to fool it. The authors study ways to make these systems more robust by uniting different training approaches under one framework. They explain why adding a certain type of penalty helps make the system more resistant to attacks. They also propose a new training method that automatically adjusts how much this penalty affects learning. Finally, they test their idea on several tasks to show it works.
What this means in practice
- •For robotics developers: Train control policies that maintain performance despite disturbances or attacks in continuous control scenarios.
- •For autonomous vehicle engineers: Develop driving algorithms that automatically tune robustness to sensor noise and adversarial environmental changes.
Authors
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Abstract
Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.