AI summaryⓘ
The authors address a problem in large language model (LLM) systems where the system looks like it's doing well overall but actually makes serious mistakes in how it gets to those results, which is dangerous in settings like car cockpits. They create CockpitHAT, a new method that improves how we find the cause of failures by looking at more than just text, including how pieces of information depend on each other and multiple channels like vehicle and environment data. They also introduce CockpitBench, a dataset to test failure diagnosis with safety importance levels. Their method outperforms previous text-only approaches in accurately identifying which part or step caused errors. Their work shows that considering dependencies, multiple sources of information, and safety risk helps better diagnose failures in real-world LLM multi-agent systems.
LLM multi-agent systemsCorrectness Collapsedependency structuremulti-channel evidenceISO 26262 ASILfailure attributionembodied systemsconfidence-weighted consensussafety-aware evaluationdiagnosis benchmark
Authors
Wei Wang, Shuanghe Liu, Zhu Zhuo, Jiaqi Zhong, Xiaozhao Zhao, Xiaojie Zuo, Jie Su
Abstract
LLM multi-agent systems suffer from Correctness Collapse, where high task-level accuracy conceals severe process-level failures. This is especially hazardous in safety-critical embodied settings such as automotive cockpits, where lexically correct utterances may trigger dangerous physical operations. Existing attribution methods rely on text traces alone, missing dependency structure, multi-channel evidence, and safety-aware evaluation. We introduce CockpitHAT, a hierarchical attribution framework that replaces positional windows with dependency-distance thresholds from interaction DAGs, integrates multi-channel evidence via an embodied adapter, and applies a safety-uplift to high-risk failures during confidence-weighted analyst consensus. We further release CockpitBench, a benchmark of 212 annotated failure traces spanning dialogue, vehicle-state, environmental, and memory channels, each labeled with ISO 26262 ASIL severity via three-expert consensus. On the public Who&When benchmark, CockpitHAT achieves agent-level / step-exact accuracies of 77.9% / 37.8% on the Hand-Crafted split and 86.5% / 46.0% on the Algorithm-Generated split, surpassing the text-only SOTA ECHO by up to 17.6 / 16.7 points. On CockpitBench, it attains 78.3% agent-level and 38.2% step-exact accuracy. These results establish dependency-aware, multi-channel, risk-calibrated attribution as an effective paradigm for reliable failure diagnosis in real-world embodied LLM multi-agent systems.