Proximal policy optimization improves risk-managed stock trading returns
PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading
Artificial Intelligence
Summary
Trading using reinforcement learning often faces challenges balancing making profits and avoiding big losses. The authors propose PPO-HRAP, a method that mixes a smart decision-making algorithm with market condition signals to act more sensibly during volatile times. Their approach performed better than simply holding stocks, showing higher returns and smaller losses on the SPY index from 2020 to 2022. It also worked well on other stock indexes in limited tests, though it still trades quite frequently and needs more testing on different assets.
What this means in practice
- •For quantitative trading teams: Use PPO-HRAP to build trading systems that balance profit goals with controlling losses through regime-aware signals.
- •For financial risk managers: Incorporate regime-aware reinforcement learning strategies to reduce maximum drawdowns in equity portfolios during volatile markets.
- •For algorithmic asset managers: Develop new trading products that blend learned policies with volatility signals to improve risk-adjusted performance.$Commercial implications: Enables creation of risk-aware trading algorithms that can be marketed to asset management firms seeking improved drawdown control.
Authors
Duong Hien Chi Kien, Thanh Trung Huynh
Abstract
Reinforcement learning for trading often struggles to balance upside participation with drawdown control. Profit-only policies can collapse toward passive long exposure on upward-drifting assets, while aggressively risk-penalized rewards can become too defensive during volatile periods. This paper proposes PPO-HRAP, a hybrid regime-aware policy that combines Proximal Policy Optimization with an interpretable regime prior. The agent observes both market features and portfolio-state variables, receives a reward combining portfolio log return, VIX-conditioned drawdown-increase penalty, target-exposure deviation, and turnover cost, and executes a blended action between the PPO actor output and a regime-derived target exposure. On the held-out 2020-2022 SPY test window, PPO-HRAP achieves 27.62% total return, 8.48% annualized return, 0.6447 Sharpe ratio, 0.8588 Sortino ratio, and 0.4592 Calmar ratio, while reducing maximum drawdown from 34.10% for Buy and Hold to 18.47%. Across five SPY seeds, PPO-HRAP remains stable with mean total return $0.2725 \pm 0.0109$ and mean Sharpe ratio $0.6219 \pm 0.0565$. Single-run cross-asset tests on QQQ and DIA further show that the proposed method ranks first on total return and Sharpe ratio for all three reported assets. These results suggest that blending learned actions with a volatility-aware regime prior is a practical way to improve risk-adjusted trading behavior, although the current policy still incurs high turnover and cross-asset robustness beyond SPY remains limited to single-run evidence.