Efficient offline learning of risk-aware decision policies
Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
Machine LearningArtificial Intelligence
Summary
Making decisions that account for risk is important in many fields but learning good risk-aware policies from past data without new experiments is hard. The authors study mathematical ways to measure risk and develop algorithms that learn risk-sensitive strategies from fixed datasets efficiently. They prove how well their methods work and also show the fundamental limits of learning in this setting. Their work improves understanding of how risk preferences influence the difficulty of offline learning.
What this means in practice
- •For autonomous vehicle developers: Train driving policies from logs that balance safety risk without costly or dangerous real-world trials.$Commercial implications: Enables safer autonomous driving systems trained from offline data that handle risk appropriately.
- •For financial algorithm designers: Use fixed historical market data to develop trading strategies sensitive to different risk attitudes without online testing.
Authors
Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu
Abstract
Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension $d$. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.