Multi-agent collaboration often underperforms single-agent systems when serving users
Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings
Artificial IntelligenceMultiagent Systems
Summary
When many AI agents work together to help people, it is not always better than just one agent. The authors tested this idea using conversations about airlines, retail, and telecom. They found that using multiple agents often made things worse or no better than one agent, especially when dealing with a fixed simulated user. This suggests that for practical use, just one agent might be better unless multi-agent setups prove their worth in similar real scenarios.
What this means in practice
- •For chatbot developers: Decide when to use single-agent or multi-agent models for user conversations based on direct performance evidence rather than assumption.
- •For customer service platform engineers: Optimize AI resource spending by preferring single-agent designs unless multi-agent systems show clear performance gains in matched compute and domain tests.
Authors
Ethan Yang, Xiangcheng Shi
Abstract
Multi-agent systems (MAS) built from language models are deployed on the premise that collaboration beats a single-agent system (SAS), yet when it pays is unclear. The most systematic study to date gives a capability-saturation rule: coordination helps while the SAS is weak and hurts once it is strong, with the crossing near 45% single-agent accuracy. That rule was derived on autonomous benchmarks, where no user takes part. We test it where agents now work: in conversation with a user under a written policy, on $τ^2$-bench, the dual-control benchmark. On its airline, retail and telecom domains we compare the stock SAS with a centralized MAS built from the same model for six models from three families, over about 7,950 simulated conversations, with one completion-token ledger per episode per system. The study was pre-registered with a public change log; the MAS is reported as built, with one validity bar unmet (its planner never saw the tool schemas). Under a fixed simulated user no cell shows a MAS benefit: the MAS is worse in 12 of the 13 valid cells and tied in the thirteenth, and the fitted gain is negative across the observed capability range (weighted mean -28 points, 95% CI [-44, -12]). On the two cells where the rule predicts a benefit, the MAS loses by 13 and 22 points, and without a budget ceiling it at best draws level at 5.5 times the SAS's token spend. The one gain appears on the cell nearest 45%, under a second simulator. Whether the gain declines with capability is unresolved (slope -0.41, 95% CI [-1.51, +0.68]), because model-by-domain effects dominate. The MAS does not detectably reduce wrongful database writes; it converts other failures into budget exhaustion and inaction. For agents that serve people, the SAS is the default; a multi-agent design must show compute-matched evidence from the setting itself. We release all trajectories, ledgers, prompts and code.