Dependency based method improves error tracing in AI agents

DeFA: Dependency-Guided Failure Attribution for LLM Agents

Artificial Intelligence

Summary

Large language model (LLM) agents sometimes make mistakes far from where the problem actually started, making it hard to find the real cause. The authors introduce DeFA, a method that links the steps an agent takes to see which steps caused errors and where those errors spread. It breaks long tasks into smaller parts to keep track of what happens overall while focusing on details. DeFA helps identify who made the mistake, exactly when it happened, and what kind of error it was, improving the ability to fix these problems and improve AI performance on future tasks.

What this means in practice

  • For ai system developers: Improve debugging of LLM agents by precisely tracing errors to their source steps and agents in complex tasks.
  • For multimodal ai engineers: Apply DeFA to identify errors in AI workflows that involve images and videos alongside text inputs.

Authors

Bo Deng, Xinlei Zheng, Yi Wei, Kang Zhou, Chongyang Tao, Renzhao Liang, Xuanren Chen, Lifan Guo, Chi Zhang

Abstract

Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.