Approximate queueing model helps autoscale large language model servers
An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling
Performance
Summary
Serving large language models (LLMs) quickly is important but challenging when demand changes. The authors developed a simple yet effective way to predict how long it takes for such servers to start generating answers and to produce each word. Their method uses three main measurements related to the model and hardware to estimate delays under varying workloads. They tested this model on real systems with different LLMs and showed it helps adjust server capacity automatically to keep delays low. Compared to existing methods, their approach misses fewer latency targets and uses resources more efficiently.
What this means in practice
- •For cloud infrastructure teams: Automatically adjust the number of large language model inference servers to meet latency targets as user request loads change dynamically.
- •For ai platform engineers: Improve prediction of start and per-token delay timings in serving pipelines to optimize throughput and resource allocation for popular LLMs.
Authors
Vishakha Ramani, Asser N. Tantawi
Abstract
Performance models of LLM servers support both latency evaluation and the design of controllers for autoscaling against service level objectives (SLOs) and for inference optimization. We model the multiplexed execution of prefill and decode operations with a tractable, approximate queueing model under Markovian assumptions. Three parameters characterize a model-accelerator pair, namely a baseline per-iteration overhead, a per-token compute cost, and a per-token key-value (KV) cache access cost. The model combines a mean-value analysis of per-iteration work with a state-dependent Markov chain for batch occupancy to predict mean time to first token (TTFT) and inter-token latency (ITL). We validate these predictions against measurements and show that the three parameters can be estimated from observed latencies. Over a grid of input and output lengths and arrival rates spanning light to moderate load, the relative error of the average ITL is about 5% for Llama-3.1-8B and 8% for Qwen2.5-14B running on an H100 GPU, and the corresponding TTFT errors are 14% and 16%. We then implement an autoscaling controller that uses the model to adjust inference-server replica counts as the workload changes. On an OpenShift cluster of H100 GPUs it tracks a fourfold load ramp under both latency targets, missing one in 7 of 127 control cycles, and its in-loop predictions carry median errors of at most 5% for TTFT and 9% for ITL. A decode-throughput analyzer from an existing autoscaler, which takes no latency target, misses 27 of 128 cycles under the same controller and load while provisioning 4% and 28% fewer replicas.