Reassessing difficulty labels reveals slow learning in reinforcement learning prompts
Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR
Artificial Intelligence
Summary
This paper looks at why some prompts in reinforcement learning seem very hard for AI to learn. The researchers found that what were called 'hard' prompts do improve, just more slowly than others, and that the way these difficulty labels are measured is often unreliable. They created a method to check how stable these difficulty ratings are and tested some earlier explanations about why certain prompts remain hard to learn. Their results show that while some prompts do learn slowly, the way we label and explain this difficulty needs more careful measurement.
What this means in practice
- •For machine learning engineers: Improve the reliability of prompt difficulty labels when training reinforcement learning models for better performance evaluation.
- •For ai quality assurance teams: Use sampling-based frameworks to quantify the stability of difficulty measurements in reinforcement learning evaluation processes.
Authors
Chandak Chakma, Syed Nazmus Sakib, Nafiul Haque, Shifat E. Arman
Abstract
Reinforcement learning with verifiable rewards (RLVR) has become an important approach for improving reasoning during post-training. Recent work suggests that some difficult prompts remain resistant to learning even when they occasionally produce correct solutions. We revisit this unlearnability phenomenon and find that the affected prompts do improve, at roughly one third of the learnable rate, while the difficulty-defined set used to study them is much less reproducible than expected. These difficulty labels are estimated from a limited number of sampled responses. Combining them across seeds can further change which prompts are selected instead of simply reducing measurement noise. We develop a sampling-based framework for quantifying this instability and determining how much evaluation is required for difficulty assignments to reproduce reliably. We also revisit the gradient-similarity evidence proposed to explain unlearnability and show that part of the observed separation arises because difficult prompts provide fewer correct rollouts from which their gradients can be estimated. Matching this sample count weakens the gradient difference but does not remove it. Overall, the slow-learning phenomenon survives our reanalysis, while both the prompts used to define it and the evidence used to explain it require more careful measurement.