Calibrated routing improves efficiency in disaggregated llm serving systems
Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
Artificial Intelligence
Summary
When computers use big language models (LLMs) to answer questions, they often split the work between different GPUs to be faster. The authors studied a smart way to decide which GPU handles each request by predicting how long it will take, considering things like prompt size and memory. They tested their method on real GPUs and found it works better than simple approaches like round robin or picking the least busy machine. Their method helps the system handle more requests efficiently and shows the most consistent performance.
What this means in practice
- •For cloud infrastructure teams: Optimize the assignment of incoming LLM requests to separate GPU pools to increase throughput and reduce latency in large-scale AI services.
- •For data center operators: Improve utilization in disaggregated GPU clusters by using calibrated routing to reduce hardware needs without sacrificing handling capacity.
Authors
Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
Abstract
Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces. It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.