GRPO outperforms rejection-sampling fine-tuning in low-data learning
Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning
Machine Learning
Summary
This paper looks at two ways to improve AI learning when only a small amount of data is available. They compare a method called GRPO to another called rejection-sampling fine-tuning (RFT) on a specific task involving counting. The researchers found that GRPO consistently learns better and faster than RFT using the same data and conditions. This suggests GRPO’s way of learning is more effective for this type of problem, although results are based on a synthetic task with limited experiments.
What this means in practice
- •For machine learning engineers: Choose GRPO over RFT to improve learning efficiency in low-data reinforcement learning scenarios similar to procedural tasks.
- •For ai model fine-tuning teams: Use GRPO fine-tuning on mixed-difficulty prompts to achieve better performance than standard rejection sampling when tuning base models with limited verification.
Tested on simulated data.
Authors
Lakshya Chaudhry
Abstract
Bauer et al. (2026) showed that in low-data reinforcement learning with verifiable rewards, training GRPO on mixed-difficulty prompts gives up to 5x the sample efficiency of easy-only prompts. They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too. We test this on their procedural counting task with Qwen3-4B-Base and LoRA, comparing GRPO to rejection-sampling fine-tuning (RFT) on the same prompts, verifier and token cap. Six primary arms cross the two methods with three prompt sets (Easy, Mixed, and a Curated subset of prompts the base model sometimes solves, which makes GRPO's implicit filter explicit). We use three seeds per arm. Hypotheses and falsifiers were registered before training. To rule out a poorly tuned baseline, RFT receives a 27-run hyperparameter sweep, while GRPO uses Bauer et al.'s recipe unchanged. GRPO outperforms RFT at every prompt set: by 10.9 points on held-out test problems with curated prompts, and, in post hoc comparisons with matched rollout budgets, by 8.4 and 9.2 points on the easy and mixed prompts. On test the advantage survives the least favorable treatment of truncated completions; out of distribution, truncation prevents a conclusion. Curating prompts does not help RFT (-2.7 points). Mixed-difficulty data helps RFT (+3.7 points), while for GRPO the evidence is inconclusive. A secondary iterated-RFT arm, which resamples from the current policy, recovers 3.2 of the 10.9-point gap (about 30%). These are controlled measurements on one synthetic task family and one base model with three seeds per arm.