Vesta improves scheduling for faster and smoother AI language requests
Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads
Distributed, Parallel, and Cluster Computing
Summary
AI apps that chat with large language models (LLMs) send many back-and-forth requests. Each new request builds on previous conversations, and reusing stored memory (called a KV cache) helps speed things up. But always sending requests to the same server can cause delays there, while others sit idle. The researchers created Vesta, a smart scheduler that balances where requests run, considering memory reuse, current wait times, and data transfers. This approach reduces slowdowns and makes the AI respond faster in real-world tests.
What this means in practice
- •For ai infrastructure teams: Optimize distributed LLM serving to reduce request delays and increase throughput for multi-turn AI applications.
- •For cloud platform operators: Manage resource allocation dynamically to improve utilization and responsiveness of AI workload clusters.
Authors
Lancheng Zou, Bowen Li, Wei Mao, Wenbin Ye, Lecheng Li, Kai Liu, Chao Deng, Jianchen Zhu, Bei Yu
Abstract
Agentic applications generate multi-turn sequences of LLM requests, with each new turn extending the context accumulated from prior interactions and tool calls. Prefix caching reuses the KV cache of earlier turns and avoids redundant prefill, but routing every turn to its cache-affine instance can create severe local queues while capacity remains idle elsewhere. Remote KV cache access relaxes this tension, yet migrating execution adds transfer work and is harmful when the cluster is already congested. Thus, deciding where to execute a request requires reasoning jointly about cache locality, queueing, and the current operating point of the cluster. We present Vesta, a queueing-aware scheduler for distributed agentic LLM serving. Vesta assigns each session a stable holder for its authoritative KV cache while independently selecting an executor for every request. Grounded in classical queueing theory, its objective accounts for backlog, reusable prefixes, KV cache transfer, and the queueing externality of additional service demand; it migrates execution to recover idle capacity when beneficial and favors placements that add the least prefill and transfer work as congestion rises. Vesta also migrates reusable prefixes across length buckets to avoid cold starts. On real-world agentic workloads, Vesta improves throughput by 14.0\% and reduces mean TTFT by 57.8\% over the state-of-the-art scheduler in a PD-colocated deployment. In PD-disaggregated deployments, it reduces mean TTFT by up to 18.1\% and P99 TTFT by up to 29.0\% at matched throughput.