Stabilizing the critic improves training for language model reinforcement learning
EasyPPO: Stabilizing the Critic Is Key
Machine Learning
Summary
Training large language models with reinforcement learning often becomes unstable because the component estimating future rewards, called the critic, can fail in ways that confuse learning. The authors find two key problems with the critic that cause this instability during training. They introduce EasyPPO, a modified training method that keeps the critic stable by carefully adjusting how it learns from both complete and partial experiences and by controlling noisy updates. Experiments show that EasyPPO leads to more stable training and better performance on tasks like coding, math reasoning, and search.
What this means in practice
- •For machine learning engineers: Improve stability and performance when training language models with reinforcement learning by using methods to better train the critic.
- •For software developers building ai tools: Build more reliable AI applications that use language model fine-tuning via reinforcement learning by employing stable critic training techniques.
Authors
Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao, Dacheng Li, Wenhao Chai, Mayank Mishra, Yichuan Wang, Karthik Narasimhan, Alvin Cheung, Joseph E. Gonzalez
Abstract
A key strength of Proximal Policy Optimization (PPO) is its learned critic, which uses historical trajectories collected during reinforcement learning to estimate expected returns and reduce policy-gradient variance. However, we find that the critic is also a major source of instability in reinforcement learning for large language models (LLMs). We identify two critic failure modes that destabilize PPO. First, filtering truncated rollouts from both actor and critic shifts the policy objective to reward conditioned on completion, allowing truncation to increase even as conditional reward improves. Second, heterogeneous return noise can cause high-variance prompts to dominate critic updates in finite batches. We introduce EasyPPO to address these failures. Actor-only overlong filtering trains the critic on returns from both completed and truncated rollouts. Noise-normalized critic regression weights each prompt's critic loss by the inverse standard deviation of its sampled returns, balancing noise contributions across prompts. Moderately smaller critic mini-batches confine outlier influence to fewer rollouts during gradient clipping. Across continuous-reward coding on FrontierCS, binary-reward mathematical reasoning on AIME24, and multi-turn search on Search-R1, EasyPPO remains stable throughout the full training horizon and consistently outperforms vanilla PPO, VAPO, and HL-Gauss PPO. Its best validation scores show relative gains of 14.89%, 2.28%, and 9.47% over PPO, respectively.