Pipeline parallelism boosts large model prefill throughput by reusing prefixes
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
When large language models process text, they often split requests into parts and handle them in stages to be faster. But making sure these stages stay busy without wasting time is tricky when each stage caches data separately. The authors developed WavePP, a system that carefully coordinates the reuse of parts of previous text across all stages while new requests are still running. This approach greatly improves the number of requests the model can process at once, nearly tripling throughput in some cases.
What this means in practice
- •For machine learning engineers: Increase the throughput of prefill operations in large language models when deploying pipeline parallelism to serve many concurrent requests.
- •For cloud service operators: Reduce latency and improve server utilization for deployed large language model inference using prefix reuse across pipeline stages.
Authors
Aaryam Sharma
Abstract
Pipeline parallelism can improve prefill throughput by processing multiple request chunks concurrently across different stages of the model. However, keeping the pipeline fully utilized requires efficient scheduling and request preparation. In systems where stages retain and evict cache state independently, a local cache hit does not guarantee that the same prefix can be reused across the pipeline. Here, coordination overhead can impede request admission cadence and thus reduce overall throughput. In this paper, we present WavePP, a prefill runtime built on top of TensorRT-LLM that addresses these challenges by overlapping request admission with pipeline execution. WavePP asynchronously finds a prefix that can be reused across all stages, protects the cached state, and reserves space for the remaining input while earlier requests continue to execute. It subsequently plans the chunk sizes of each request dynamically to maximize pipeline fill. Each stage then completes the local preparation before executing the request. In the same system and pipeline topology, WavePP improves TensorRT-LLM's prefill throughput in 37 of 40 tested settings on GLM 5.2 and MiniMax M2.7. At concurrency 128 with high cache reuse, these changes increase throughput by factors of 2.91 and 2.02, respectively. Across 28 Kimi K3 settings, WavePP also has the highest measured throughput in all 18 settings at concurrency eight or higher, compared with tensor/expert-parallel and pipeline-parallel baselines from TRT-LLM, SGLang, and vLLM.