Trading Human Curation for Synthetic Augmentation in RLVR
2026-06-02 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how to efficiently create lots of training tasks for teaching language AI models with clear rewards, which usually needs a lot of time and effort from humans. They try using a small number of carefully made tasks and then automatically generate more variations to see if this can replace some human work. Their tests show that these automatic task versions work almost as well for training and can save money. They also measure how much cheaper synthetic tasks are compared to human-made ones across different scenarios. Overall, their approach keeps the AI performing well while reducing the cost of creating training tasks.
Reinforcement LearningVerifiable RewardsAgentic Language ModelsTask AugmentationReward FunctionsTraining TasksSynthetic DataCost-Adjusted Trade RateGeneralizationPrompt Engineering
Authors
Akshansh <last>, Leonardo Rosa Rodrigues, Michael Korostelev, Youssef Hassan, Mark E. Whiting
Abstract
The supply of high-quality training tasks is a central bottleneck for reinforcement learning from verifiable rewards (RLVR) on agentic language models. Each task requires a sandboxed setup, a prompt, and a hand-authored reward function, and only tasks that pass a quality bar produce useful training signal. Hand-curation at this quality bar does not scale economically to the task counts effective RL training requires, and the substitution rate between automatically generated task variants and human-authored ones is not yet established. We investigate using pre-specified, gate-filtered augmentations of a small hand-authored base as a substitute for additional human curation during RLVR. We formalize the cost-adjusted trade rate $ρ_{\text{cost}}$ between augmented and human-authored tasks, measure it through a controlled ablation across training corpora with varying augmentation share, and characterize the end-to-end economics of the augmentation pipeline. Substituting augmented content for additional human-authored tasks retains aggregate held-out generalization on a ten-benchmark suite spanning code, instruction following, reasoning, and multi-turn agentic function-calling. The cost-adjusted trade rate $ρ_{\text{cost}}$ between gated synthetic and human-authored RLVR tasks stays in $[1.4\times, 11.6\times]$ across the plausible $c_{\text{human}}/c_{\text{aug}}$ range.