ThinkPrior improves early prompt learning efficiency in reinforcement learning
ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
Machine LearningArtificial Intelligence
Summary
When teaching computers to learn tasks through trial and error, some groupings of attempts don't help the learning process because they all succeed or all fail, producing no useful feedback. The authors found that a lot of effort is wasted on these unhelpful groups early in the learning process. They introduced ThinkPrior, which uses an external pre-learned estimate to predict how difficult prompts will be before trying them. This approach reduces wasted attempts early on without changing the overall learning results. It reallocates effort more effectively rather than reducing the total work done.
reinforcement learningprompt selectiongroup relative policy optimizationreward-advantagezero-advantage groupsdifficulty priorBeta posteriorrolloutverifiable rewardsoffline initialization
Authors
Tommy Sha, Skylar Zhai, Siqi Zhao
Abstract
In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.