Neural policies trained to handle rare extreme risks in reinforcement learning
Deep catastrophic-risk-aware reinforcement learning
Machine Learning
Summary
Estimating very rare but severe risks in reinforcement learning is challenging because few extreme examples exist to learn from. The authors extended a statistical theory called Extreme Value Theory (EVT) to help neural network policies better predict and handle these rare events. They tested this approach first in a controlled setup and then in financial trading scenarios where extreme market crashes are rare but impactful. Their method led to policies that performed similarly in normal conditions but reduced losses in extreme cases, suggesting EVT can improve training when extreme examples are limited.
What this means in practice
- •For quantitative finance teams: Develop hedging strategies that better withstand rare extreme market crashes by training neural policies using EVT methods for tail risk.
- •For machine learning engineers: Improve robustness of reinforcement learning policies in domains with scarce extreme event data by optimizing a differentiable EVT-based tail-risk objective.
Authors
Carlos Pérez Octavio Mendoza, Frédéric Godin
Abstract
Estimating extreme risk in reinforcement learning becomes increasingly difficult as the confidence level increases, since only a small number of observations contribute to empirical estimates of Conditional Value-at-Risk (CVaR). In this paper, we extend an Extreme Value Theory (EVT) approach for CVaR optimization to high-dimensional neural-network policies. Using a peaks-over-threshold representation and method-of-moments estimation, we obtain a differentiable tail-risk objective and show that, conditional on the selected threshold, its finite-sample gradient can be computed directly through automatic differentiation. We first validate the approach in a controlled experiment with a known optimal solution and then investigate its performance in a deep-hedging problem with rare extreme market movements. Our results show similar hedging behavior for EVT- and empirical-CVaR-trained policies under normal conditions, while EVT-trained policies exhibit lower hedging losses under the extreme-scenario evaluation across the training sample sizes considered. These results suggest that EVT can provide a practical alternative for training neural policies when the observations relevant to extreme risk are scarce.