Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training
2026-08-10 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how to better choose tasks for training large language models using reinforcement learning. They found that tasks which seem equally easy to solve right now can actually differ in how much the model improves by practicing them more—this is called task learnability. To predict learnability before full training, they created TrajVal, a simple tool that estimates how beneficial continuing training on each task would be. Experiments showed that using TrajVal to pick tasks makes training more efficient than picking tasks randomly and also works well combined with other scheduling methods.
reinforcement learninglarge language modelstask samplingtask learnabilityreward trajectoryprobe-based estimatorTrajValdata efficiencyonline schedulingpost-training optimization
Authors
Ting Zhou, Zhenqing Ling, Daoyuan Chen, Qianli Shen, Yilun Huang, Ying Shen, Yaliang Li
Abstract
Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy. However, tasks with similar current solvability can still differ substantially in how positively they respond to further training. We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. By analyzing per-task reward trajectories, we find that learnability is reproducible across independently sampled training contexts and predictive of downstream utility. To make this signal practical before training begins, we propose TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations. TrajVal can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online schedulers. Experiments on mathematical and logical reasoning benchmarks across multiple model scales show that TrajVal improves data efficiency over uniform sampling and provides complementary gains when combined with online scheduling methods.