Battery storage dispatch improved by cost-aware reinforcement learning methods
Cost-Aware Reinforcement Learning with Action Masking and Projection for Battery Energy Storage Dispatch under Suppressed-Spread Market Shifts
Machine LearningArtificial Intelligence
Summary
Managing battery energy storage to make money from shifting electricity prices can be tricky because the price differences are shrinking. The authors tested a smart computer program that uses forecasts and rules to decide how to charge and discharge a battery while avoiding unsafe actions. Their method improved profits over simpler approaches in some cases but still didn't beat some traditional control methods. They showed how separating economic advice from safety checks helps keep the battery running within safe limits without guaranteeing perfect outcomes.
What this means in practice
- •For energy storage operators: Use cost-aware reinforcement learning with action masking to improve battery dispatch decisions under variable electricity prices while respecting operational constraints.
- •For smart grid system designers: Incorporate layered safety enforcement separate from economic decision layers for more reliable battery management in electricity markets with low price volatility.
Authors
Kuanlin Chen, Chen-Wei Kuo, Cheng-En Ou
Abstract
Battery energy storage system (BESS) dispatch must preserve operational feasibility while declining price spreads reduce the margin available to pay for cycling. We study a proximal policy optimization (PPO) controller whose pre-selection physical action mask and emergency projection are separated from a causal, forecast-informed economic advisory. All forecast-dependent methods receive the same causal 24-step forecast and grid-side settlement. Across five PPO seeds, advice-on net profit is 30.59 and 18.04 USD per 336-hour T1 and T2 window, versus 36.77 and 22.94 USD for proxy-cost MPC; PPO remains below this reference in both periods. Advice raises T2 profit from 16.45 to 18.04 USD while reducing throughput, but is immaterial in T1. On disjoint weekly blocks, PPO is stable under daily, weekly, and blended seasonal forecasts, weakens under persistence, and remains below proxy-cost MPC. Paired diagnostics localize changes to the observed 5-10 USD/MWh regime with mixed SoC-dependent effects. An M0-M6 ablation shows that mask removal sends thousands of infeasible requests to projection, while removing both physical layers exposes ramp violations. The evidence separates economic screening from feasibility enforcement without claiming formal safety, lifecycle-optimal aging, or RL dominance.