Memory-aware predictive scheduling improves large language model serving latency
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
Artificial Intelligence
Summary
Large language models running on cloud servers can experience delays because the length of the generated outputs is unknown when requests arrive. The authors propose a framework called MAPS that predicts output lengths early using the user's device and uses these predictions to schedule tasks more efficiently in the cloud. This helps balance workloads and reduces waiting times, leading to faster responses when using language models. The approach was tested on real workloads and showed significant latency improvements compared to existing systems.
What this means in practice
- •For cloud infrastructure teams: Improve scheduling of large language model tasks to reduce server latency and balance memory usage.
- •For mobile app developers: Enable better handling of language model requests by predicting output lengths early on user devices to speed up cloud responses.
Authors
Tiancheng Zhang, Yulin Chen, Yunfeng Zhao, Shaoyuan Huang, Cheng Zhang, Xiaofei Wang
Abstract
The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling, incurring negligible latency overhead. To handle generation uncertainty, MAPS applies uncertainty-aware calibration to derive output-length upper bounds with target coverage, enabling safe scheduling decisions. Building on these bounds, MAPS employs a hierarchical global-local scheduling strategy to mitigate inter-decoder queue buildup and intra-decoder head-of-line blocking. Extensive experiments on two real-world workloads and two LLMs show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6 and tail latency by up to 84.8.