Multi-turn agents learn to avoid and fix early mistakes
PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
Artificial Intelligence
Summary
When AI agents make mistakes early during tasks, these errors often cause more problems later, making it hard to complete the task. The authors found that some early errors, called pivotal mistakes, can often be fixed if the agent receives guidance soon after. To improve this, they created PivotOPD, a training method that teaches the AI both to avoid these critical mistakes and to recover from them quickly. This approach improved the agents’ success rates on several tasks and different AI models.
What this means in practice
- •For language ai developers: Improve multi-turn language agents to better avoid early critical errors and recover when mistakes happen, increasing task success.
- •For software engineering teams: Raise success rates of AI assistants in software engineering tasks by training with pivotal mistake recovery strategies.
Authors
Yinghui He, Yapei Chang, Khushi Bhardwaj, Daniele Molinari, Tugrul Konuk, Jan Kautz, Ali Hatamizadeh
Abstract
On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories. However, in multi-turn interaction, an incorrect action changes the states the student encounters later, so errors compound across turns. In preliminary experiments across three Qwen3 models (8B to 235B), we find that more than half of the failed rollouts contain a pivotal mistake, an action that moves the agent farther from completing the task, and this mistake typically occurs early. These pivotal mistakes often remain recoverable: guiding the model for only a few turns after the pivotal turn can restore task success. We therefore propose PivotOPD, an on-policy distillation framework that jointly trains the student to prevent pivotal mistakes and to recover from the states they create. At each pivotal mistake, a teacher model provides a gold action and then names a recovery action at each of the next few turns. Preventive distillation uses the gold action with reverse KL to steer the student away from the pivotal mistake, while recovery distillation uses the recovery actions with forward KL to transfer recovery behaviors that the student rarely samples. Against 13 baselines on ALFWorld, WebShop, and Search-based QA, PivotOPD achieves the strongest average performance for both Qwen3-1.7B and Qwen3-8B students, improving over the strongest baseline on ALFWorld by +5.5% with the 1.7B student. The gains also transfer to another model family on the software engineering domain, where PivotOPD raises the resolve rate of a Nemotron-3.5 student on SWE-Bench Verified by +3.2%. Project page: https://research.nvidia.com/labs/lpr/pivotopd/