Tool builds causal graphs to diagnose agent failures effectively
From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis
Artificial Intelligence
Summary
When AI agents make mistakes in complex tasks, it can be hard to figure out why. The authors created a tool called CEG-Agent that separates different kinds of problems like anomalies, errors, and failures, and connects them in a clear cause-and-effect graph. They also made a benchmark called CEG-Bench to test how well this works, and found their tool matches expert human judgments closely. This helps people understand and fix why AI agents fail more reliably.
What this means in practice
- •For ai system developers: Diagnose and trace the root causes of failures in complex AI agent executions using causal error graphs.
- •For software reliability teams: Improve failure attribution in automated workflows by integrating causal diagnosis tools with detailed error taxonomy.
Authors
Shu-Xun Yang, Yidong Wang, Zhuoer Feng, Bosi Wen, Jiayi Gui, Dayong Yang, Wenbo Yu, Haoke Zhang, Jie Tang, Cunxiang Wang
Abstract
LLM-driven agents are increasingly deployed in complex applications, where long agentic traces make failures difficult to diagnose. Existing trace diagnosis methods often conflate anomalies, errors, and failures, making diagnostic targets ambiguous; they also lack structured modeling of how causally relevant errors propagate and amplify into final task failures, resulting in unreliable failure attribution. To address these problems, we propose CEG-Agent, a tool-augmented agentic framework for causal diagnosis of agentic traces. Specifically, CEG-Agent introduces an explicit taxonomy of anomalies, errors, and failures, and constructs Causal Error Graphs (CEGs), a unified typed representation that links execution events, diagnostic nodes, and failure outcomes through causal relations. To evaluate causal trace diagnosis, we further construct CEG-Bench, a fully agent-annotated benchmark with high-confidence, consensus-derived CEG annotations obtained through an Adversarial Agentic Adjudication Protocol (AAAP). We validate the resulting annotations against an expert-curated human gold set, which shows close agreement with the automatic annotations. Experiments on CEG-Bench demonstrate that CEG-Agent achieves state-of-the-art performance under both semantically relaxed and structurally exact evaluation criteria. Our code is publicly available.