BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
2026-08-31 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors introduce BCPPO, a new method to train decision-making models that better manage rare but costly events. They use multiple cost-prediction networks to identify uncertain predictions and turn this uncertainty into a penalty that helps the model avoid risky choices. Their technique improves the balance between achieving high rewards and reducing potential high costs without changing how the cost predictors learn over time. Testing across many scenarios showed BCPPO generally outperforms others by having better returns while keeping risk low. This approach keeps the final model simple by only requiring the trained policy for use.
Proximal Policy Optimization (PPO)Monte CarloConditional Value at Risk (CVaR)Cost-prediction networksTemporal Difference (TD) learningPolicy gradientBachelier formulaRisk-sensitive reinforcement learningDisagreement penaltySafe reinforcement learning
Authors
Dongsheng Hou, Yanqiao Chen, Yuhan Rui
Abstract
Expected-cost constraints can still permit rare, high-cost events. Monte Carlo conditional value at risk (CVaR) gradients can be noisy at high confidence, whereas critics that model an outcome distribution add complexity. We propose BCPPO (Bachelier-Inspired Constrained Proximal Policy Optimization), a proximal policy optimization (PPO) method. Separately initialized cost-prediction networks (critics), trained with random sample masks, produce disagreement that marks predictions sensitive to which state-action regions occur in the training data and to critic training. A Bachelier formula for the expected amount above a reference level converts this disagreement into a smooth policy-update penalty. Gradients from this penalty do not alter the critics, so temporal-difference (TD) critic learning is unchanged. A saturation-aware controller adjusts the mean-cost penalty and stops accumulated error from growing while that penalty is clipped. Deployment retains only the policy network. The disagreement penalty is neither a tail-event probability nor a guaranteed error bound, and it provides no safety guarantee. Across 175 runs with shared tasks, costs, budgets, training steps, and evaluation seeds, no comparator attains both higher mean return and lower mean CVaR than BCPPO in any task. On Push1, BCPPO has no lower return and no higher CVaR than every comparator, with at least one strict gain. These results support a practical balance among reward, caution around cost predictions that vary across trained critics, and policy-only deployment.