Error propagation modeling improves fault finding in multi agent AI systems
Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
Artificial Intelligence
Summary
Multi-agent AI systems, where multiple AI agents work together, can sometimes fail, but it's hard to figure out which agent or step caused the problem. The authors introduce EMFA, a new way to track how errors flow and persist across interactions, helping pinpoint the exact agent and step responsible. This method uses structured error modeling and tests fixes virtually to find the key mistake. It performed better than previous methods on a known benchmark.
What this means in practice
- •For ai system developers: Identify precise error sources in multi-agent AI workflows to improve system debugging and reliability.
- •For software testing teams: Enhance automated debugging tools by integrating error propagation models for multi-agent interactions.
Authors
Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu, Xu Zhang, Zheming Zhuang, Dawei Feng, Bo Ding, Huaimin Wang
Abstract
LLM-based multi-agent systems (MASs) are increasingly used to solve complex tasks through coordinated reasoning, tool use, and interaction with external resources. However, attributing failures in such systems remains challenging because the observed outcome often does not directly reveal the error responsible for the failed execution. In this work, the attribution target is the decisive error, defined as the agent--step pair whose correction would recover the failed execution. Existing approaches largely identify suspicious steps without explicitly modeling how errors propagate across interactions or persist in unresolved loops, making decisive errors difficult to distinguish from downstream failure symptoms. We propose \textbf{E}rror-Propagation \textbf{M}odeling for \textbf{F}ailure \textbf{A}ttribution (\textbf{EMFA}). EMFA constructs a structured representation of the failed trajectory, models both cascading propagation and persistent interaction loops, and uses propagation-aware candidate screening followed by counterfactual verification to identify the decisive agent--step pair. On the Who\&When benchmark, EMFA achieves state-of-the-art step-level attribution accuracy and remains competitive at the agent level. It improves the previous best step-level results by 3.45 and 4.40 percentage points on the Hand-Crafted and Algorithm-Generated subsets, respectively.