Papers for

real-time ai platform engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Adaptive chunk sizing improves prompt handling in large language models

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Abstract: Continuous batching improves large language model (LLM) serving throughput, but long prompt prefills can delay decode iterations and violate inter-token latency objectives. Chunked prefill mitigates this interference, yet its chunk size is normally fixed: small chunks protect decode latency but repeatedly pay launch overhead, while large chunks improve prefill efficiency but create latency spikes. We introduce SLOWeave, an online scheduling method that selects the largest prefill chunk predicted to finish before the earliest active decode deadline. The decision requires no workload-specific chunk-size tuning and is computed by a logarithmic-time search over a monotone iteration-cost model. We prove that, whenever a decode-only iteration is feasible and the cost predictor is accurate, SLOWeave maximizes immediate prefill progress among decisions that preserve every active request's next-token deadline. We evaluate the method in a reproducible event-driven simulator and an iteration-level GPU runtime across chat, mixed-context, long-context, and bursty workloads. Under a 25ms time-per-output-token objective, SLOWeave improves goodput over the strongest fixed-chunk baseline by 39% on mixed requests and 38% on long-context requests. Under a stricter 10ms objective, the gains rise to 3.3$\times$ and 2.4$\times$, respectively. These results isolate adaptive chunk sizing as a useful serving primitive and provide an implementation-ready controller for integration with iteration-level LLM runtimes.

Mon 7 SeptComputation and Language
The gist
Large language models take some time to get ready for generating text, which can delay responses when handling many requests quickly. The authors introduce SLOWeave, a way to split this preparation into smart chunk sizes based on how quickly the model needs to respond to each request. This approach balances speed and efficiency so the model works faster without making users wait longer at important moments. Testing shows it delivers more results on time compared to fixed chunk sizes.
Open → 2609.07883v1