Predictive bandit curriculum improves reinforcement learning training efficiency
Predictive Bandit Curriculum Reinforcement Learning via Gradient-Based Sample Group Discovery
Machine Learning
Summary
Training computer programs to learn from experience can be slow and inefficient. The authors found that grouping training examples based on how they affect learning and then predicting how useful these groups will be helps train better programs faster. They developed a method that clusters samples by their influence on learning and forecasts which groups will be most helpful next. This approach leads to smarter use of data and faster improvement across different tasks.
What this means in practice
- •For reinforcement learning engineers: Improve training speed and data use in reinforcement learning models through adaptive curriculum design based on sample gradient behavior.
- •For ai software developers: Integrate predictive sample grouping to enhance model learning efficiency in applications requiring reasoning tasks.
Authors
Liang Xu, Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Wei Lin, Guojun Yin, Shuhui Wang
Abstract
Curriculum learning, which organizes training samples according to their learning priority, has been widely adopted in reinforcement learning to improve training efficiency. However, existing methods commonly construct curriculum units using predefined data properties and estimate their utility by simply aggregating a single proxy signal from recent training history.Such designs may group samples with different optimization effects and fail to anticipate how their utility evolves in relation to the policy. We propose Predictive Gradient-based Bandit (PGB) Curriculum, which first discovers optimization-aware learning units by clustering projected sample-level gradients. PGB then employs a lightweight temporal predictor to forecast the future utility of each unit and converts the predictions into a Boltzmann sampling distribution for adaptive data allocation. Experiments across multiple model scales and reasoning benchmarks show that PGB consistently outperforms standard GRPO, difficulty-based curricula, and history-based scheduling strategies, demonstrating improved data utilization and training efficiency.