Reinforcement learning method improves solving hard prompts faster
What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR
Machine LearningArtificial Intelligence
Summary
Some tasks in reinforcement learning are hard to solve and often stay unsolved for a long time before suddenly becoming solved, like a light switch turning on. The authors study what causes this sudden improvement and find that unsuccessful attempts right before success behave differently from completely hopeless ones. They introduce a new method, NAPO, that uses this insight to better learn from failures and help the learning process jump from zero to success faster. Their method showed clear improvements on challenging math reasoning tests.
What this means in practice
- •For machine learning engineers: Improve efficiency in training reinforcement learning agents on tasks with abrupt success transitions by using driver-guided negative learning optimizations.
- •For automated theorem proving teams: Enhance automated proof search by steering policy updates according to early signs in failed attempts, accelerating discovery of valid solutions.
Authors
Rui Zhu, Yumou Liu, Zhenzhe Zheng, Fan Wu
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.