Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how to reduce wear on heat pump compressors caused by turning on and off frequently. They used two reinforcement learning methods, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), adding a penalty for compressor cycling in the controller's reward. SAC learned to keep the compressor running smoothly without turning it off, which reduced discomfort greatly but slightly increased energy cost. PPO, on the other hand, kept switching the compressor on and off more than usual. This shows that the choice of learning algorithm affects how well the system balances comfort, cost, and equipment wear.
Reinforcement LearningHeat PumpCompressor CyclingSoft Actor-Critic (SAC)Proximal Policy Optimization (PPO)Markov Decision ProcessHydronic Heat PumpControl RewardThermal ComfortBOPTEST Emulator
Authors
Faizan Ahmed, Aniket Dixit, James Brusey
Abstract
On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.