Adaptive scheduling boosts large language model response speed and efficiency
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Computation and Language
Summary
Large language models (LLMs) process text in batches to work faster, but handling long initial input sequences can slow down generating each new word. The authors present SLOWeave, a method that smartly chooses how much of the initial input to process at once, so it finishes before the next word must be produced. This approach avoids delays caused by fixed chunk sizes and improves how many responses are finished on time. In testing, their method made LLMs work much faster and more reliably across different types of text workloads without needing manual tuning.
large language modelprefillbatchinglatencythroughputchunk sizeschedulingworkloaddecode iterationdeadline
Authors
Siyu Song, Qi Bai, Jinbo Hao, Kai Li, Chenchen Wang, Jiayu Sun
Abstract
Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.