Kvcompass improves large model response speed across global servers
KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving
Distributed, Parallel, and Cluster Computing
Summary
When big AI models generate text, they reuse parts of their memory to speed up future answers, but this reuse usually happens only in one location. The authors found that sharing this memory with servers in other regions is tricky because the memory pieces move around and change over time. They created KVCompass, a system that quickly finds where the right memory pieces are stored worldwide so requests can be sent there, making responses faster and saving computing power. Tests showed KVCompass significantly reduces delays and handles more requests efficiently compared to other methods.
What this means in practice
- •For cloud service providers: Optimize large language model serving across globally distributed data centers by sharing cached computations to reduce response delays and resource use.$Commercial implications: Enables cloud providers to offer faster and more efficient AI model services worldwide by sharing expensive computation caches across regions.
- •For network infrastructure engineers: Design request routing that balances load and network distance by quickly discovering where cached AI model data resides, enhancing overall latency and throughput.
Authors
Shiyi Ling, Fei Fang, Chen Qian
Abstract
KV cache reuse is one of the most effective ways to reduce LLM inference cost, but today its benefits are largely confined to a single cluster. When a provider serves the same model across geographically distributed regions, each region may repeatedly prefill the same popular prefixes, wasting GPU time and increasing time-to-first-token (TTFT). Exploiting reuse across regions involves discovering which worker currently holds a reusable KV cache for each request's prefix. We show that this KV cache discovery problem is structurally harder than conventional object lookup. Unlike hash-placement mechanisms such as DHTs, where an object's location is a deterministic function of its key, KV caches are created on the workers that serve requests, and their holder sets change with subsequent requests and KV cache evictions. KV cache discovery therefore calls for decoupling KV cache identity from placement. We present KVCompass, a discovery plane for Internet-scale KV cache sharing. The discovery index supports variable-length prefix lookup with constant-time local table reads at the edge, allowing each request to be forwarded to the worker that best balances KV cache reuse against load and network distance. Evaluation includes a 64-worker geo-distributed simulation and a 14-worker three-continent real-machine testbed. Evaluation shows that KVCompass achieves $1.3-14\times$ lower P99 TTFT than the baselines, up to $3.7\times$ lower P99 end-to-end latency, up to $4.7\times$ the throughput, and a peak load factor within 2% of the Oracle floor.