Paired rollouts help reduce noise but not always learning speed in group reinforcement learning
Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?
Machine Learning
Summary
When teaching AI agents by comparing how they perform on the same tasks, random events can make it hard to see which changes really help. The authors studied a method called paired rollouts that links randomness across comparisons to reduce noise in reward estimates. They found that this method can lower some types of variance but doesn't always make the AI learn faster. Their experiments showed some improvement in final results but not consistent gains in learning speed. This work clarifies when pairing helps and when it doesn’t in group-based AI training.
What this means in practice
- •For machine learning engineers: Use paired rollout techniques to reduce reward noise when training large language model agents facing environment and grader randomness.
- •For robotics developers: Incorporate event-keyed noise pairing in reinforcement learning systems controlling robots that experience hardware faults or inconsistent evaluations.
Authors
Nazmus Sakib
Abstract
Group-relative reinforcement learning compares rollouts of the same prompt, but independent environment noise can obscure these comparisons. We study paired rollouts, which share an event-keyed noise schedule within each group while preserving each rollout's marginal distribution. Pairing removes the between-schedule component of reward-contrast variance, but need not reduce gradient variance. For one-sided grader noise, we derive an exact condition for reduction and give a counterexample in which reward contrasts improve while gradient variance increases. A controlled study trains a 2B tool-use agent under tool faults and grader flips, with three seeds per design. The protocol was registered with a disclosed, previously completed pilot. Under tool faults, pairing improves final noisy-test success by +5.1 percentage points on average, with all three seed differences positive, but misses the registered learning-curve criterion. The criterion is also missed under grader flips: the validation-AUC difference is +0.003 (95% interval [-0.029, +0.033]). A gradient probe on eight distinct checkpoints from two fault-trained trajectories finds lower mean-centered covariance traces under both noise types: 21 to 30% for grader flips and 40 to 63% for tool faults. These finite-sample measurements support the variance mechanism without establishing a general learning-speed benefit. The results distinguish improving reward comparisons, reducing estimator variance, and improving learning.