SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created SwarmBench, a new test to measure how well groups of AI language agents work together, looking at things like accuracy, cost, and how smoothly they coordinate. They found that current AI models vary a lot in how well they manage these group tasks. To help improve this, they introduced SwarmExp, a technique that helps AI agents learn from past experiences to perform better. Overall, their work aims to make AI agent teamwork more effective and easier to evaluate.
large language modelmulti-agent systemagent swarmbenchmarkorchestrationaccuracyefficiencyexperience replaySwarmBenchSwarmExp
Authors
Jinshan Gao, Zhuoran Jin, Tianyi Men, Kang Liu, Jun Zhao
Abstract
Large language model-based multi-agent systems are evolving from fixed interaction topologies toward dynamically orchestrated Agent Swarms. However, existing benchmarks are still largely based on single-agent or general-purpose agent tasks, making it difficult to systematically evaluate key orchestration capabilities. We propose SwarmBench, a benchmark that evaluates model performance from multiple perspectives, including accuracy, efficiency, cost, and process quality. Experimental results show that current models exhibit substantial differences in orchestration capability. These differences are reflected not only in final accuracy, efficiency, and cost, but also in the overall quality of the orchestration process itself. Based on these findings, we further propose SwarmExp, a simple yet effective method based on experience extraction and experience replay, which consistently improves the orchestration performance of large language models.