AgentWeave improves distributed agent workflows with smarter scheduling
AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition
Distributed, Parallel, and Cluster Computing
Summary
Workflows that combine calls to large language models and tools can be slow when distributed across many devices because some parts take longer than others, leaving devices idle. The authors propose AgentWeave, a system that breaks long tasks into smaller pieces so short and new tasks don’t get stuck waiting behind long ones. By doing this, AgentWeave keeps devices busier and balances processing more efficiently. Their experiments show this approach speeds up work and makes responses faster compared to existing methods.
What this means in practice
- •For distributed system engineers: Build faster agent workflows by decomposing and scheduling tasks to reduce idle time and latency in multi-device setups.
- •For cloud service operators: Improve responsiveness and throughput of AI-powered services that rely on complex agent workflows across distributed infrastructure.
Authors
Kaibin Guo, Pengtu Li, Zicong Hong, Zhiyuan Fang, Wuhui Chen, Rachid Guerraoui, Anne-Marie Kermarrec
Abstract
Agent applications are evolving into workflows composed of large language model (LLM) calls and tool invocations, termed modules. Existing distributed agent systems typically adopt a disaggregated architecture, deploying different modules on separate devices. While this enables pipeline parallelism, imbalanced module latencies introduce severe pipeline bubbles that limit throughput. A natural alternative is an aggregated architecture that co-deploys all modules across all devices, eliminating bubbles by keeping every device continuously active. However, this introduces request blocking: short requests are forced to progress in lockstep with longer ones in the same batch, and newly arriving requests must wait until the entire ongoing batch completes the full workflow. To address this, we propose AgentWeave, an efficient distributed serving system for agent applications built on the aggregated architecture. Its core mechanism, flow decomposition, decomposes long agent requests into fine-grained execution units while preserving agent semantics, allowing short and newly arriving requests to proceed without being stalled by long-running ones. Complementing this, a flow-adaptive scheduler dynamically balances the throughput gains of decomposition against its scheduling overhead. Experiments across four representative agent applications demonstrate that AgentWeave improves throughput by 1.81x, reduces average latency by 2.45x, and reduces P90 latency by 2.04x over state-of-the-art baselines.