Rollout agreement improves reasoning accuracy in AI policy training
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Machine LearningComputation and Language
Summary
This paper looks at how AI models reason through problems by examining differences in their correct answers. The researchers found that when AI reasoning steps agree more with each other, the answers tend to be less flawed or confusing. They developed a new reward method called RoAR that encourages AI models to focus on agreement in reasoning steps, improving their accuracy on math problems. This method works without needing extra human help or complicated tools and shows better answers compared to previous methods.
What this means in practice
- •For machine learning engineers: Improve AI reasoning accuracy in verification-dependent tasks by incorporating rollout agreement rewards during model training.
- •For automated math problem solvers: Increase the reliability of AI systems solving math problems by using agreement-based reward signals to reduce spurious reasoning.
Authors
Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung
Abstract
Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood. We investigate this question through a blind paired analysis of a model trained with Group Relative Policy Optimization (GRPO), comparing the highest- and lowest-agreement correct-answer traces for the same problem. Across 126 pairs, low-agreement traces have more spurious reasoning than high-agreement traces (35% vs. 22%), including internal inconsistencies and misuse of problem conditions. Based on these findings, we propose RoAR (Rollout-Agreement Reward), which adds reasoning-level agreement to GRPO's rewards for verified-correct rollouts. RoAR computes this signal from existing on-policy rollout groups, requiring no human-written rationales, additional sampling, or separately trained process evaluator. On five mathematical reasoning benchmarks, the 1.5B GRPO-RoAR model reaches 55.5% mean pass@1 under zero-shot evaluation using 7K training examples, exceeding GRPO and GRPO-CARE by 6.0 and 5.5 percentage points. A separate blind audit of 100 problems answered correctly by all four 1.5B models finds spurious rates of 18% for GRPO-RoAR, 32% for GRPO, 27% for GRPO-CARE, and 37% for DRA-GRPO. Our findings identify reasoning validity as an important consideration when using rollout differences to guide learning. Our code is available at https://github.com/ldi-kyunghee/RoAR.