Soft actor critic training issues tracked by early value size growth
Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
Artificial IntelligenceMachine Learning
Summary
Training AI agents to learn from experience can run into problems when the agent updates too often between new data. The authors studied two issues that cause failures in a popular learning algorithm called Soft Actor-Critic: the representation collapses or the value estimates grow too large. They found that value growth early in training, measured by the size of the Q-values, predicts which runs will fail later, but representation collapse does not. Detecting this early can save computation by stopping bad training runs in advance, and adding normalization changes these dynamics and affects final performance.
What this means in practice
- •For reinforcement learning engineers: Stop training runs early by monitoring the early growth of Q-values to save compute resources on expensive continuous control tasks.
- •For robotics development teams: Improve reliability of Soft Actor-Critic training by using early warnings from value magnitude growth to avoid wasted training on unstable models.
Authors
Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao
Abstract
Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude $|Q|$. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or $|Q|$ explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's $\log_{10}|Q|$ has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's $|Q|$ sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.