Deep reinforcement learning improves risk-sensitive decision making
Deep Reinforcement Learning with Buffered Quantile Objectives
Machine LearningArtificial Intelligence
Summary
Choosing safe or risky actions is hard when small changes can cause big differences in outcomes. The paper introduces a method called Deep-BQRL that helps computers learn better decision-making strategies by focusing on a smooth measure of risk instead of a single risky outcome. This method uses neural networks and learns from experience without needing a complete model of the world. The researchers tested it on problems like when to sell an asset and navigating slippery paths, showing it can adjust behavior based on different risk levels.
What this means in practice
- •For financial analysts: Implement asset-selling strategies that adjust to different financial risk levels more reliably using model-free reinforcement learning.
- •For robotics developers: Train robots to handle uncertain and slippery environments by learning safer policies sensitive to different risk thresholds.
Authors
Mohammad Alipour-vaezi, Sajad Khodadadian
Abstract
Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.