Mentor system helps AI learn better from failures in real time
MENTOR: Learning from Failure with On-Policy Pedagogical Intervention
Artificial Intelligence
Summary
When AI systems learning complex tasks suddenly stop getting helpful feedback, they can get stuck and stop improving. The authors show that instead of ignoring these failures, we can use their clues to help AI understand what went wrong. They created MENTOR, a helper AI that watches the learning process live, spots the cause of failure, and gives useful hints to guide the AI on its next try. This approach fits easily into existing training methods and helps AI models do much better on tasks like math and coding.
What this means in practice
- •For ai developers: Improve training of AI models on difficult tasks by integrating real-time mentors to prevent learning stalls.
- •For automated code generation teams: Enhance code generation models by dynamically guiding learning agents away from failure states during training.
Authors
Guanlan Yue, Hang Yu, Xiaoyong Huang, Ruipeng Li, Wei Zhang, Bingchang Liu, Rui Wang
Abstract
Reinforcement Learning from Verifiable Rewards (RLVR) is a key driver for Large Reasoning Models, but it is vulnerable to "zero-reward collapse", a training failure where a sudden increase in task difficulty causes the learning signal to vanish. We argue that the conventional approach of discarding these failed trajectories is wasteful, as they contain valuable diagnostic information about the model's cognitive gaps, a bright side that can be leveraged for recovery. Existing solutions, however, often regress to static, "off-policy-style" interventions like pre-defined hint libraries or perform post-hoc refinement, failing to provide the adaptive, forward-looking guidance needed to escape the collapse. To address this, we propose a paradigm shift towards on-policy pedagogical intervention and introduce MENTOR, a multi-agent framework that acts as a live mentor. MENTOR diagnoses the root cause of failure in real-time and dynamically generates targeted, instructional prompts to guide the learning agent's next attempt. As a plug-and-play module, MENTOR integrates seamlessly into existing policy optimization pipelines like GRPO without modifying the training infrastructure. Extensive experiments on math and code generation show that MENTOR effectively mitigates zero-reward collapse, outperforming state-of-the-art baselines. Notably, it achieves massive average relative gains of up to 78.3% on Llama-3.2-3B-Instruct over its base model.