Teacher corrections improve AI learning on hard logic tasks
PIVOT: Progressive Intervention via Online Teacher Correction
Computation and Language
Summary
Teaching AI to solve difficult reasoning problems can be tricky because feedback is often sparse or unclear. The authors show that giving AI both detailed step-by-step hints and occasional interruptions with correct guidance helps it learn better. They introduce a method called PIVOT that combines these two types of teacher corrections. This approach improves AI performance on challenging logic puzzles and math reasoning tests compared to previous methods.
What this means in practice
- •For ai system developers: Improve training of AI models for complex reasoning tasks by integrating teacher corrections into reinforcement learning.
- •For educational software engineers: Design intelligent tutoring systems that combine trajectory interventions and detailed feedback to guide student problem solving.
Authors
Zhihao Yang, Boyuan Zheng, Puhui Tan, Yukun Chen, Kuan Zhang, Longze Chen, Jiaming Li, Renke Shan, Lu Wang, Xianpei Han
Abstract
Reinforcement learning from verifiable rewards (RLVR) is limited by sparse outcome-level feedback on hard reasoning tasks: when all rollouts in a group fail, rewards provide no learning signal. Stronger teachers can provide additional guidance, but this raises a central question: how can such signals be incorporated into RLVR to support successful rollouts while remaining learnable for the student? We distinguish two complementary mechanisms: trajectory-level intervention, which improves access to reward-bearing trajectories, and token-level supervision, which provides dense targets to guide student learning. Our controlled experiments show that either mechanism alone can remain insufficient, motivating their combination. We therefore introduce PIVOT, which inserts teacher-generated corrections into student rollouts and applies token-level KL supervision to these segments alongside the RLVR objective. Beyond their combination, we systematically analyze where to intervene and how to supervise, yielding a recipe for effective guidance. Across hard logic games and mathematical reasoning benchmarks, PIVOT consistently outperforms existing methods, nearly doubling mean avg@8 on logic games from 19.5% for the best-performing baseline to 35.6%.