Fairness-Aware and Latency-Controllable Scheduling for Chunked-Prefill LLM Serving
2026-06-08 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingPerformance
AI summaryⓘ
The authors studied how to better manage tasks when large language models handle many requests at once, which can cause delays and unfair waiting times. They found that current methods, which serve requests in the exact order they arrive with fixed work amounts, often lead to slow response and unfairness. To fix this, they created a new scheduling approach that gives higher priority to tasks that have waited longer and adjusts work dynamically based on expected timing. Tests on GPUs showed their method reduces average waiting time and helps avoid very slow responses. Their code is available for others to use and try.
large language modelschunked-prefill executionscheduling policyfirst-come first-served (FCFS)latencyhead-of-line blockingrequest starvationaging-based schedulinglatency predictionprefill concurrency control
Authors
Haoxin Liu, Jiayi Wang, Yueshen Xu, Rui Li
Abstract
As large language models (LLMs) are increasingly deployed with highly heterogeneous workloads, chunked-prefill execution has emerged as a mainstream serving architecture. Balancing scheduling fairness and latency stability in such environments is critical; otherwise, severe head-of-line blocking and request starvation will degrade user experience. However, existing systems rely on rigid First-Come, First-Served (FCFS) policies and static token budgets, leading to fairness degradation and unpredictable latency jitter. To address these issues, we propose a fairness-aware and latency-controllable scheduling framework for chunked-prefill LLM engines. Specifically, we design a lightweight aging-based scheduling policy that dynamically calculates priorities using accumulated waiting time and remaining prefill work. Furthermore, we develop Latency-Prediction-Based Request Scheduling (LPRS) and Active Prefill Control (APC) to replace static budgets with target-time constraints and actively regulate prefill concurrency. We evaluated our scheduling framework on NVIDIA GPUs and Ascend accelerators using real-world workloads. Results show the aging policy reduces mean end-to-end latency by over 10\% compared to FCFS. Moreover, LPRS and APC significantly reduce P99 tail latency and suppress prefill fragmentation, confirming that the structural prefill control and the temporal latency constraints are fundamentally complementary. All codes have been released in Github.