Curriculum difficulty progression improves training in group reinforcement learning
A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO
Artificial Intelligence
Summary
Training AI models often involves showing them data that gets harder over time, known as curriculum learning. The paper studies a method called GRPO, which picks medium-difficulty training examples to learn efficiently. The authors find that gradually moving from medium to slightly harder examples helps the AI learn better than sticking to medium difficulty. However, pushing to very hard examples can hurt performance. This suggests a balanced approach to gradually increasing difficulty works best.
What this means in practice
- •For machine learning engineers: Improve reinforcement learning training efficiency by implementing staged curricula that adjust sample difficulty dynamically.
- •For ai system integrators: Optimize learned model accuracy in group reinforcement learning tasks by tuning the progression of data difficulty during training.
Tested on one dataset.
Authors
Ronghan Li, Fei Chen, Mingyu Lu
Abstract
Curriculum learning typically organizes training samples from easy to hard. In GRPO, medium-difficulty samples are attractive for online selection because they are more likely to yield informative group-relative learning signals, and prioritizing them can improve training efficiency. However, it remains unclear whether persistently selecting such samples also leads to the strongest final model performance, or whether training can benefit from progressing toward harder samples. Meanwhile, sample difficulty changes as the policy evolves, so fixed difficulty estimates can become misaligned with the current policy. We therefore re-estimate sample difficulty under the current policy and, inspired by the easy-to-hard principle of curriculum learning, design and evaluate a staged curriculum that progressively shifts the active difficulty band toward harder samples. Experiments on GSM8K show that persistent medium-difficulty selection does not achieve the strongest observed performance. Across three random seeds, progressing to a moderately harder difficulty band improves test accuracy to $0.6591 \pm 0.0077$, compared with $0.6449 \pm 0.0024$ for the fixed-medium curriculum. However, further progression to a harder difficulty band reduces accuracy to $0.6405 \pm 0.0053$. These results show that staged difficulty progression can outperform persistent medium-difficulty selection, but its benefit is non-monotonic. Effective GRPO curricula can therefore benefit from progressing beyond medium difficulty, while avoiding excessive progression toward overly difficult samples.}