OR Else: A Differentiable Trust Region for Policy Optimization
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study a new method called Output Reset (OR) for improving training in large language models compared to the existing clipped objectives used in PPO and GRPO algorithms. PPO-OR improved the average training reward compared to PPO-clip, but with more variability across trials. For GRPO, the OR variant did not raise average rewards but showed more stable optimization behavior. Overall, OR changes the way optimization behaves but its benefit depends on the specific algorithm and setup. The results are based on reward model scores during training, not human evaluations.
PPOGRPOOutput Reset (OR)clipped surrogate objectiveslarge language modelsgeneralized advantage estimation (GAE)reward modelreinforcement learning with human feedback (RLHF)
Authors
Chinmay Rane, Kanishka Tyagi, Michael Manry
Abstract
PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training. PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space; the advantage sign determines the update direction, and a token contributes zero direct OR residual after crossing the favorable margin. We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages, using \texttt{Llama-3.2-1B-Instruct} on Anthropic \texttt{hh-rlhf} with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score $0.305$ higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At $G=2$, the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.