OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
2026-08-13 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors address the challenge of efficiently managing GPU resources for serving large language models in the cloud while meeting strict user performance goals. Instead of scaling the entire model at once, they propose OpScale, which scales parts of the model (called operators) individually to better handle variable workloads. This fine-grained approach helps avoid delays or wasted GPU capacity. Their tests show OpScale can save GPU usage and power or increase throughput within the same cost constraints.
Large Language ModelsGPU ClustersAutoscalingService Level Objectives (SLOs)Operator-level ScalingResource ManagementInference WorkloadsCloud ComputingModel ServingThroughput
Authors
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang, Jiarong Xing, Haoran Qiu
Abstract
Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LLMs). Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.