Locating hidden mistakes improves trust in long autonomous AI tasks
Locating Hidden Failures Makes Long-Horizon Agents More Reliable
Machine Learning
Summary
When AI systems perform long tasks on their own, we usually only check if they succeed at the end. This doesn't tell us where they made mistakes or if they caused any problems during the task. The authors studied thousands of task runs and categorized many types of mistakes. They found that once an AI makes a mistake, it usually doesn't notice or fix it, which can cause hidden damage even if the task finishes successfully. They created a special tool that can find these hidden mistakes better than experts, helping improve the reliability of long-running AI systems.
What this means in practice
- •For software engineering teams: Use failure localization to identify hidden errors in automated software development tasks and improve debugging processes.
- •For system reliability engineers: Integrate failure detection tools to monitor autonomous systems and reduce risk of unnoticed damage during long operations.
Authors
Salman Rahman, Yubin Kim, Mihir Parmar, A. Ali Heydari, Genglin Liu, Simon A. Lee, Weizhi Zhang, Arian Hosseini, Ahmed A. Metwally, Yuzhe Yang, Baharan Mirzasoleiman, Xin Liu, Pavel Izmailov, Saadia Gabriel, Mark Malhotra, Shwetak Patel, Daniel McDuff, Hamid Palangi
Abstract
As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed. An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. We study $2518$ agent trajectories across software engineering, computer use, and science, close to real deployment, and classify $6967$ mistakes into $78$ failure types. Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. Yet Scout, a $4$B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.