Large language models hold hidden clues to pinpointing failures
A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems
Machine Learning
Summary
Figuring out exactly where a group of AI agents goes wrong on a task is tricky and usually needs a lot of human work. The authors found that while AI models often don’t explicitly say which step caused a failure, they actually have the information hidden inside. They introduced a simple way to uncover this hidden knowledge to better identify failure points without needing extra labeled examples. This helps improve how we can automatically find mistakes in AI systems working together.
What this means in practice
- •For multi-agent system developers: Identify specific failing steps within collaborating AI agents to speed debugging and improve system reliability.
- •For customer support platform engineers: Enhance error diagnosis in AI-driven support tools by automatically tracing faults to precise decision points in conversation agents.
Authors
Wendy Zheng, Liang Wu, Yinhan He, Mayank Darbari, Jundong Li, Liangjie Hong
Abstract
Agentic systems have demonstrated strong capabilities on complex tasks by enabling multiple interacting large language model (LLM)-based agents to collaborate and execute subtasks in parallel. However, this collaborative structure introduces new challenges for reliably evaluating and diagnosing system failures, particularly failure attribution: tracing a system's execution to identify the agent step responsible for an incorrect final outcome. Because manually performing this task is time- and labor-intensive, recent work has turned to LLMs as judges to automate the process, referred to as automated failure attribution (AFA). These approaches assume that a sufficiently capable model can infer the failing step directly from an interaction transcript. In practice, however, direct prompting often performs poorly, which motivates follow-up work that explores various ways to improve AFA performance, including alternative prompting strategies and inference-time interventions. In this paper, we examine a more fundamental question: whether this weak behavioral performance reflects a genuine capability limitation. Using linear probes applied to internal activations, we uncover a significant \emph{knows-but-won't-say gap:} models encode sufficient information to attribute a failure correctly but fail to explicitly output the responsible step. Building on this finding, we propose \emph{grafting}, a simple manual intervention that exploits knowledge already encoded in the model to improve AFA performance without requiring ground-truth labels at inference time. Extensive evaluations across multiple LLMs and benchmark settings demonstrate that grafting consistently outperforms direct-prompting approaches, underscoring the potential of leveraging latent model knowledge for more reliable failure attribution. Our code is available at https://github.com/wendyzheng22/mi4afa.