Group relative rl improves learning from saturated data with negative rollouts
Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL
Machine LearningComputation and Language
Summary
When training AI systems that learn from groups of example answers, sometimes all the answers get the highest scores, making it hard for the AI to learn what’s better or worse. The authors looked at ways to recover useful learning signals even when all answers seem equally good. They found that making the AI generate some high-quality but incorrect answers helps by creating examples with worse scores, making the training more informative. This approach helps reuse data that would otherwise be useless, improving learning performance.
What this means in practice
- •For ai model trainers: Improve reinforcement learning on data where all outputs have top rewards by generating high-quality negative examples for better training signals.
- •For automated reasoning developers: Recycle previously saturated reasoning datasets by introducing carefully crafted incorrect solutions to enhance model learning.
Authors
Ziyuan Yang, Yike Wang, Shangbin Feng, Yulia Tsvetkov
Abstract
Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.