Multi-agent language models underperform compared to matched single passes

More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs

Artificial IntelligenceComputation and LanguageMachine LearningMultiagent Systems

Summary

When many AI language models work together and communicate, it might seem they do better than just one model thinking once. But this paper shows that comparing multi-agent setups to single tries mixes up how much thinking, how many tries, and how the models cooperate affect results. The authors analyzed hundreds of experiments and found multi-agent models usually do not improve accuracy compared to carefully matched single tries or equal computing effort. They explain that coordination benefits only happen if agents have different information or make different kinds of errors.

What this means in practice

  • For ai software engineers: Avoid computational waste by choosing single-pass models over multi-agent setups when evidence is shared and error diversity is low.
  • For automated customer support teams: Design chatbot ensembles with diverse information inputs or verification steps to potentially gain from multi-agent communication.

Authors

Seyed Mahmoud Sajjadi Mohammadabadi

Abstract

Evaluations of multi-agent language models often compare communicating architectures against a single greedy pass. This practice confounds coordination with sample multiplicity and inference compute. We synthesize nine primary empirical evaluations comprising 516 unique multi-agent configurations organized along a resource vector R with six dimensions and a baseline ladder with four categories. Against baselines with a single pass (Category A), multi-agent accuracy differences are negative overall and near zero when omitting the largest contributing study. When evaluated against baselines matched on sample count (Category B) or resource budget (Category C), multi-agent systems average -0.042 in accuracy, and multilevel random effects modeling on the log odds scale remains negative across sensitivity checks. Multi-agent accuracy equals or falls below the baseline in 74% of these comparable contrasts, and study means are nonpositive in almost all evaluated studies. Across studies reporting both baselines, moving from a greedy pass to a sampled baseline shifts accuracy differences by -0.038 on average. Because communication adds no information when agents share identical evidence, positive returns require partitioned evidence, external verification, or error diversity across models. We provide a reviewer checklist and accounting utilities.