Error-targeted learning reduces agent mistakes in complex tasks
MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors
Machine LearningArtificial IntelligenceComputation and Language
Summary
When computer agents try to complete long tasks by following instructions, they sometimes make repeated mistakes that cause failure. The authors propose a method called MiRL that first finds common mistakes the agent makes and then trains it specifically to avoid those mistakes by trying again from the error points. This targeted training helped the agent reduce its errors significantly and perform better on new tasks. The method also uses fewer training attempts than previous approaches, showing it can improve complex agents more efficiently.
What this means in practice
- •For software developers: Improve automated command-line agents to fix recurring errors more efficiently for complex task completion.
- •For autonomous system engineers: Enhance training processes for intelligent agents handling multi-step, instruction-based workflows by focusing on fixing specific failure modes.
Authors
Runchu Tian, Zaid Khan, Zun Wang, Justin Chih-Yao Chen, Mohit Bansal
Abstract
Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.