Parallel tool-use agents face challenges in error recovery and localization
ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents
Machine LearningSoftware Engineering
Summary
Many AI agents use tools to complete tasks, but they often fail to spot and fix errors along the way. The authors created ParaRecover, a large set of test cases that show different types of mistakes agents make when working with multiple tools at once. They also made a scoring system to check how well agents can find and fix these mistakes during their work. Tests show even top AI models struggle with error propagation and replanning. Their scoring method can help improve agents' ability to reflect and recover from mistakes.
What this means in practice
- •For software developers: Assess and improve AI assistants that rely on multiple tools by measuring their ability to detect and recover from errors during task execution.
- •For automation engineers: Build more reliable multi-step automation workflows by using ParaRecover to identify weak points in error handling and recovery of tool-using agents.
Authors
Bowen Guan, Zhentao Yin, Yanming Shen
Abstract
Existing agent benchmarks mainly evaluate final task success or tool-call correctness, providing limited insight into whether agents can reliably diagnose and recover from intermediate execution failures. This limitation becomes particularly critical in multi-turn parallel tool-use scenarios, where errors may propagate across dependent branches and trigger cascading failures. We introduce ParaRecover, a process-level benchmark for evaluating error localization and recovery in multi-turn parallel tool-use agents. Built upon a fine-grained taxonomy of 14 error types covering planning dependencies, tool selection, and argument matching, the benchmark comprises 10,626 instances spanning two difficulty levels. To enable finegrained, process-oriented evaluation, we further propose the SDE rubric, which measures structural integrity, diagnostic reasoning, and evolutionary strategy during agent execution.Experiments across more than ten mainstream LLMs reveal that even state-of-the-art models still struggle with multi-turn error propagation,implicit tool-use failures, and precise replanning. Moreover, we demonstrate that the SDE rubric provides effective supervision signals for improving agents' reflective recovery capabilities. Our data and code are available at https://github.com/gbw206/ParaRecover.