Cross-model autoscaling cuts latency in shared large language model serving
Cross-Model Autoscaling for Shared LLM Serving
Distributed, Parallel, and Cluster Computing
Summary
When multiple large language models (LLMs) run on the same GPUs, they compete for resources and must quickly respond to user requests. Existing systems treat each model separately, causing delays and inefficient resource use. The authors present TRE, a system that measures and balances the workload fairly across models, reallocating GPU resources where needed to keep response times low. Experiments show TRE significantly reduces delays compared to previous methods.
What this means in practice
- •For cloud infrastructure teams: Manage shared GPU clusters more efficiently to reduce response delays for multiple concurrent LLM services.
- •For ai service providers: Improve latency guarantees for multi-model LLM hosting platforms serving varied user demands on fixed hardware budgets.$Commercial implications: Enables more reliable and efficient multi-model LLM hosting products for customers needing predictable performance.
Authors
Xin Zhang, Xianyan Xie, Zhen He, Xijin Yin, Xingtong Lin, Bangbo Liang, Zequn Cheng, Peihao Huang, Guo Chen
Abstract
Multi-model LLM serving is moving toward shared MaaS clusters, where co-hosted models compete for a fixed GPU budget while each model experiences time-varying demand and must satisfy its own latency SLO. Existing LLM autoscalers remain largely model-local: their signals expose local runtime activity or delayed latency outcomes, and their scale-up or provisioning decisions do not directly determine how shared capacity should be allocated across competing models. We present the Token-service-share Rebalancing Engine (TRE), a control-plane framework for hot-switched multi-model LLM serving. TRE introduces Token Service Share (TSS), a calibrated, demand-normalized signal that estimates effective token service per active or queued request and yields a comparable health score across heterogeneous models and SLO classes. Guided by TSS, TRE coordinates bounded receiver--donor capacity movement under a fixed GPU budget: it separates fast rescue from slower rebalancing and incrementally reallocates active replicas toward models with the largest calibrated service deficits. We implement TRE on a Kubernetes-based hot-switch serving stack without modifying the inference scheduler. Across seven LLM serving traces, TRE reduces P95 end-to-end latency by 11.9--79.0\% and P99 latency by 12.5--72.6\% compared with a state-of-the-art KV-cache-based reactive autoscaler running on the same hot-switch runtime. The gains hold on both targeted stress probes and production-derived conversation/code traces, where TRE reduces P95/P99 latency by 50.8/63.7\% and 79.0/72.6\%, respectively. These results show that effective hot-switched autoscaling requires not only fast replica actuation, but also calibrated service-deficit signals and coordinated cross-model capacity arbitration. Our code and artifacts are available at https://github.com/zxzx9898/Token-service-share_Rebalancing_Engine.