Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
2026-08-17 • Hardware Architecture
Hardware ArchitectureDistributed, Parallel, and Cluster ComputingMachine Learning
AI summaryⓘ
The authors studied how to better manage different priority levels for serving large language models (LLMs) to users with varying needs. They improved a system called Llumnix by allowing more than just two priority levels and tested it with a simulator named Vidur. Their experiments showed that using four priority tiers balances performance and cost well, speeding up responses significantly compared to other methods. They also found that even with ten priority levels, the system works efficiently without major slowdowns. This work helps ensure different users get appropriate service quality based on their priority.
Large Language Models (LLMs)Service-Level Objectives (SLOs)Priority SchedulingLlumnix SchedulerLoad BalancingInference SimulatorLatencyAuto-scalingHierarchical SchedulingPrefill Phase
Authors
Anders Vestrum, Arya Raeesi, Hanna Roed
Abstract
Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing. Llumnix introduced a dynamic, migration-capable multi-instance scheduler for LLM inference that achieves load balancing, defragmentation, prioritization, and auto-scaling through a unified "freeness" metric. However, Llumnix's priority model is restricted to two levels (high and normal), an abstraction too coarse to express the richer SLA classes common in production deployments. In this work, we extend Llumnix's priority model to support an arbitrary number of tiers and evaluate the effects of this extension under three realistic priority distributions (uniform, Gaussian, enterprise) using Vidur, a high-fidelity LLM inference simulator. We implement per-tier headroom with exponential decay, tier-aware dispatch ordering, and the full Llumnix migration pipeline inside Vidur's hierarchical scheduling framework. We compare our extended scheduler against INFaaS (global routing baseline), vLLM, Orca, and Sarathi-Serve (per-replica baselines), sweeping priority levels from 1 to 10. Our experiments demonstrate that four priority tiers yields the best cost-effectiveness tradeoff, achieving prefill mean speedups of up to 8.3x and end-to-end P99 speedups of up to 3.1x over INFaaS with cost-per-latency improvements of 46 to 68%, while preserving strong SLO differentiation across tiers. We further show that the system sustains these gains at 10 priority levels without tail latency collapse, with overhead concentrated in the prefill phase.