Adaptive chunk sizing improves prompt handling in large language models

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Computation and Language

Summary

Large language models take some time to get ready for generating text, which can delay responses when handling many requests quickly. The authors introduce SLOWeave, a way to split this preparation into smart chunk sizes based on how quickly the model needs to respond to each request. This approach balances speed and efficiency so the model works faster without making users wait longer at important moments. Testing shows it delivers more results on time compared to fixed chunk sizes.

What this means in practice

  • For llm service operators: Improve throughput and reduce response delays in serving large language models by using adaptive chunk sizing for prompt preparation.
  • For real-time ai platform engineers: Integrate an online scheduling method that maximizes prefill progress while meeting token latency deadlines across varied workloads.

Authors

Siyu Song, Qi Bai, Jinbo Hao, Kai Li, Chenchen Wang, Jiayu Sun

Abstract

Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.