Multi agent llm serving benchmark reveals hardware bottlenecks
MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads
Multiagent Systems
Summary
Large language models (LLMs) can work together in groups called multi-agent systems to solve complicated tasks. These systems have many steps that depend on each other and change over time, which makes it tricky to measure their performance on computers. The authors created MAServeBench, a benchmark that simulates how these agents coordinate and share information to better test how well hardware running LLMs can handle real workloads. This tool helps identify where delays and inefficiencies happen, even when hardware seems powerful.
What this means in practice
- •For llm serving infrastructure teams: Evaluate and optimize hardware setups by simulating realistic multi-agent LLM workloads that reflect complex task coordination and data exchange patterns.
- •For edge computing engineers: Analyze how multi-agent LLM tasks impact resource demands on GPU and NPU hardware to improve deployment strategies in limited-resource environments.
Authors
Shengxuan Qiu, Haochen Huang, Shuzhang Zhong, Zizhuo Fu, Zebin Yang, Jianming Tong, Meng Li
Abstract
Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible comparisons of legal release constraints at fixed work. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.