High-speed kv compression cuts delays in large language model serving

Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving

Distributed, Parallel, and Cluster Computing

Summary

Large language models often need to share key-value (KV) data quickly across computers, but sending all this data can slow things down. The authors found that just compressing the data isn't enough; the compression itself has to be very fast to really help. They created a system called system that speeds up compression by combining steps and using a better toolkit. As a result, it greatly reduces the time taken to handle KV data and improves overall processing speed with fast networks.

What this means in practice

  • For cloud platform teams: Improve throughput and reduce latency of large language model serving by integrating faster KV compression pipelines.
  • For network infrastructure engineers: Optimize interconnect usage in high-bandwidth data centers by deploying efficient KV cache compression methods to save communication time.

Authors

Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao

Abstract

KV cache compression reduces the communication volume of disaggregated large language model serving, but its practical benefit depends not only on compression ratio and model quality, but also on how efficiently the compression pipeline executes. This requirement becomes increasingly important as modern serving systems adopt high-bandwidth interconnects, where inefficient compression can offset a substantial portion of the saved communication time. This paper presents \system, a high-throughput KV cache compression data path built on top of \kvserve. \system improves compression execution in two complementary ways. First, it fuses the distribution transformation and low-bit quantization stages into a KV-aware \tilelang kernel, eliminating intermediate tensor materialization and redundant memory accesses. Second, it replaces the original lossless compression backend with an open, extensible, and searchable backend based on \lc, enabling high-performance compression operator pipelines. The fused kernel achieves up to $5.9\times$ speedup, while the new lossless backend increases compression throughput from below 90~GB/s to over 500~GB/s. Across the evaluated workloads, \system reduces job completion time by up to $2.04\times$. End-to-end evaluation shows that these optimizations allow KV compression to retain meaningful communication benefits under modern high-bandwidth network.