Reinforcement learning technique improves large language model training stability
Towards Better Training Signal: Advantage Clipped Policy Optimization
Machine Learning
Summary
Training large language models using reinforcement learning can be slow and unstable because the data used changes as the model learns. The authors propose a new technique called ACPO that stabilizes training by clipping both the importance sampling ratio and the advantage together instead of clipping just the ratio. This method makes the learning signals more stable, improving performance and training speed on math reasoning tasks. Their experiments show ACPO works better than previous methods, producing more accurate models more efficiently.
What this means in practice
- •For machine learning engineers: Improve training stability and efficiency when fine-tuning large language models with reinforcement learning on mathematical reasoning tasks.
- •For ai software developers: Develop improved reinforcement learning algorithms for language models that handle unstable gradients more robustly using clipped advantage signals.
Authors
Ruichuan Huang, Jinghan Liu, Congliang Chen
Abstract
Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.