Quantized key value transfer speeds up prompt processing by over two times

ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation

Distributed, Parallel, and Cluster Computing

Summary

Generating text using AI models often requires moving the model’s memory, called the key-value (KV) cache, between computers, which can be slow. The authors present ReKV, a way to send a smaller, compressed version of this memory and only recalculate some parts on the spot, speeding up the overall process. They tested this on powerful Nvidia GPUs and showed it can reduce transfer times by more than twice compared to sending full precision data. This method adapts depending on prompt size and network speed to find the best balance between transfer and recomputation.

What this means in practice

  • For ml infrastructure teams: Reduce latency when distributing large language model caches between servers by combining partial recomputation with compressed cache transfer.
  • For data center network engineers: Improve overall throughput and reduce data transfer bottlenecks for AI workloads by dynamically balancing recomputation and quantized data movement.

Authors

Guangjian Dong

Abstract

Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.