Crossflow improves token throughput by adapting workload balancing
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceMachine Learning
Summary
Large language models process tasks in two phases: prefill (preparing data) and decode (generating output). The demand between these two phases changes constantly, making fixed resource division inefficient. The authors developed Crossflow, a method that lets resource sharing between these phases adjust dynamically without changing server roles. This approach improves token processing speed by up to 43% under heavy load and reduces waiting time.
What this means in practice
- •For cloud infrastructure teams: Improve server resource allocation dynamically to increase processing throughput of large language model services.
- •For network schedulers: Use elastic lease management to better balance compute and transfer workload for distributed AI serving nodes.
Authors
Yi Xu, Ehsan K. Ardestani, Wenyin Fu, Martin Schatz, Krishna Malladi, Zhan Shu, Adnan Aziz, Shobhit Kanaujia, Ajit Mathews, Chunqiang Tang
Abstract
As serving capacity demand surpasses that of training, serving efficiency becomes increasingly important. Prefill-decode (P/D) disaggregation improves serving efficiency through specialization and isolation of the two phases. These benefits rest on a static partitioning. Phase demand, however, is not static. We observe that in a large LLM fleet the ratio of uncached input to output tokens has peak-to-mean ratios up to 4.7x at minute timescales, and that in a public agentic trace the hourly ratio spans a median 24.5x within a single day, while reassigning a replica takes tens of minutes. Agentic traffic sharpens the mismatch. Sizing each pool at its ninety-fifth percentile leaves up to 17% of cluster capacity unused; sizing below it converts the same imbalance into queueing and unrealized throughput. We present Crossflow, which makes this boundary elastic without changing node roles. Each decode node publishes a short-lived, revocable lease that bounds local-prefill compute, KV capacity, transfer work, and projected output. Across public and internal traces, Crossflow improves token throughput by 16.2-17.4% on geometric mean over static P/D, and by up to 43.4% at high load, while reducing mean TTFT at every evaluated point.