Multi-agent systems can fail inside yet still give correct answers
Where Do Multi-Agent Systems Fail? Evidence-Grounded Diagnosis of Collective Mechanisms
Multiagent Systems
Summary
When a group of computer agents work together to solve problems, they can sometimes break the rules about sharing and using information but still get the right answer. The authors studied how to tell if one of these rules was broken by looking at the system’s records, not just the final answer. They developed a method called diagnostic contracts to check whether evidence supports a violation of these rules. Their results show that just seeing a correct answer is not enough to know if the agents worked as they should, and tests on different systems reveal that what works for one doesn’t always work for another.
What this means in practice
- •For software reliability teams: Use diagnostic contracts to detect internal failures in agent communication beyond just checking final outputs.
- •For workflow automation engineers: Apply diagnostic evidence methods to verify that coordinated steps follow intended communication protocols in complex workflows.
Authors
Zhengye Han
Abstract
When a multi-agent system answers correctly, it is tempting to conclude that its agents shared, checked, and used information as intended. Yet a system can break one of its collective mechanisms, the rules that govern how agents route, admit, store, and act on shared information, and still return the right answer, while a wrong answer rarely reveals which mechanism failed. We ask what evidence from an execution is sufficient to conclude that a particular mechanism was violated. Our answer is a diagnostic contract, which separates what counts as a violation from which execution records can establish one, and concludes that a violation is supported, ruled out, or unknown; removing records can make this conclusion unknown but never reverse it. We test contracts for four mechanisms by replaying executions from the step where a mechanism acts, once unchanged, once with the mechanism broken, and once with it restored. Broken mechanisms often left the answer correct. An LLM diagnoser detected many more violations from internal records than from public outputs, yet with identical records a generic prompt often claimed certainty the records did not support, which prompts stating the contracts largely avoided. The contracts also applied, in narrow form, to mechanisms in independently developed systems, but a diagnostic behavior that was nearly perfect on our benchmark degraded on an independently developed workflow. A correct outcome is therefore no substitute for records of how collective mechanisms operated, and agreement on one benchmark does not show that a diagnoser transfers to another system.