Deep reinforcement learning agents tested for varied state input vulnerabilities
Beyond $L_\infty$-Norm Sets: Synthesizing Interpretable Non-Uniform Uncertainty Sets for Evaluation-Time State-Adversarial Robustness
Machine Learning
Summary
Deep reinforcement learning programs can be tricked by small changes to the information they use to decide what to do. The authors noticed that treating all parts of this information as equally important for these tricks doesn't show the whole picture, because some parts affect the decisions more than others. They created a new way to find which parts and when the program is most sensitive, helping understand its weaknesses much better. Their method works while testing and can adapt depending on the situation.
What this means in practice
- •For robotics engineers: Identify precisely which sensor readings most impact control policies under attack conditions during evaluation of robot behavior.
- •For autonomous vehicle developers: Refine robustness testing by detecting state input dimensions where driving algorithms are most vulnerable to manipulated sensor data.
Tested on simulated data.
Authors
Xulin Chen, Ruipeng Liu, Garrett E. Katz
Abstract
Deep reinforcement learning (DRL) agents are vulnerable to perturbations of their observed state inputs. Prior works often constrain state adversaries with $L_p$-norm uncertainty sets and evaluate policy performance under the resulting perturbations. However, in continuous control tasks like robotics control, different state dimensions have distinct physical meanings and influence policy behavior to different degrees, which $L_p$-norm sets fail to capture or explain. Moreover, it is important to identify when and along which state dimensions a policy is vulnerable, independently of the algorithm used to train it. In this paper, we focus on evaluation-time attacks applied to pretrained policies. We propose \textbf{P}erformance-\textbf{D}egradation \textbf{N}on-\textbf{U}niform \textbf{U}ncertainty \textbf{S}et (\method{}), an interpretable formulation of uncertainty sets for state perturbations. \method{} is state-dependent, dimension-wise non-uniform, and can be synthesized online during policy evaluation. We first formulate an ideal constrained optimization problem for \method{} computation, then introduce a surrogate constraint to make the problem tractable. Experiments across multiple continuous-control environments, policy algorithms, and state attacks show that \method{} reveals variation in policy sensitivity across states and dimensions, while its performance degradation is moderately controlled.