Summary
Reinforcement learning agents can be tricked by small changes in their inputs, which is risky in important tasks. The authors extend methods that can guarantee minimum performance under such attacks, focusing on measures that care about risk rather than just average outcomes. They develop a mathematical framework that turns this into an easier problem to solve and propose a way to improve these guarantees by adjusting how risk is handled during training. Tests on common simulation tasks and a maintenance problem show that training for risk awareness generally improves safety certificates, but too much caution can sometimes hurt performance.
What this means in practice
- •For autonomous vehicle developers: Guarantee minimum expected performance of driving policies when sensor readings are slightly manipulated by an attacker.
- •For industrial maintenance teams: Use certified risk-sensitive policies to ensure reliable machine replacement decisions despite uncertain or corrupted sensor data.
Abstract
Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $φ$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $β$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.