Rethinking the Divergence Regularization in LLM RL
2026-06-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors discuss how training large language models with reinforcement learning can be tricky because the model's behavior during training often differs from how it acts when used. Existing methods try to limit big jumps in the model’s behavior using a technique called ratio-clipping, but this can be inaccurate for rare words. A recent method, DPPO, improved this by using a mask based on how much the model’s probabilities change, but it simply ignores problematic updates instead of fixing them. To improve on this, the authors propose DRPO, which gently adjusts learning steps when the model changes too much, making training more stable and efficient.
reinforcement learninglarge language modelsoff-policy trainingtrust regionratio clippingDPPOpolicy optimizationgradient updatesstabilityregularization
Authors
Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, Tianyu Pang
Abstract
Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-region control essential for stable optimization. Mainstream methods such as PPO and GRPO approximate this control with a ratio-clipping mechanism, but the importance ratio can be a poor proxy for distributional shift in long-tailed vocabularies. Recent work such as DPPO addresses this mismatch by replacing ratio-based clipping with a divergence-based mask, yielding a trust region defined by the sampled token's absolute probability shift. However, DPPO still relies on a hard mask: once a token crosses the trust-region boundary in a harmful direction, its gradient is discarded rather than corrected. To address this, we propose Divergence Regularized Policy Optimization (DRPO), which replaces the hard mask with a smooth advantage-weighted quadratic regularizer on policy shift. DRPO preserves the same trust-region geometry as DPPO while inducing bounded, continuous gradient weights that attenuate diverging updates and provide corrective signals beyond the boundary. Experiments across model scales, architectures, and precision settings show that DRPO improves the stability and efficiency of LLM RL training.