Teacher-guided curriculum improves learning on unsolvable reinforcement tasks
Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR
Computation and Language
Summary
Sometimes AI models get stuck on problems that are too hard and don't learn anything from trying them. The authors found a way to help models learn from these tough problems by giving hints from a stronger model and slowly taking those hints away. This helps the AI get better without needing to try lots of easier problems first. Their method uses much less training data and lets models solve harder problems more reliably over time.
What this means in practice
- •For ai product developers: Create AI systems that learn complex tasks efficiently by training mainly on difficult, initially unsolvable problems.
- •For machine learning engineers: Design reinforcement learning pipelines that use teacher-guided curricula to improve model reasoning without large datasets.
Authors
Yukang Zhu, Zhen Han
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.