Caching method improves accuracy and security in shared AI models
Preserving Provenance in Shared KV Caches for LLM Serving
Distributed, Parallel, and Cluster Computing
Summary
The paper addresses a problem with how large AI language models share memory caches during use. When different requests use the same tokens but come from incompatible settings, errors and security leaks can happen. The authors studied this problem systematically and proposed a way to keep track of where each cached item came from using a 'provenance contract.' This method avoids unsafe reuse, improves accuracy, and protects timing-based leaks without slowing down the system noticeably.
What this means in practice
- •For ai platform engineers: Improve reliability and security of shared key-value caches in large language model serving infrastructures by preserving request provenance.
- •For cloud service operators: Prevent cross-tenant data leakage and accuracy loss in shared LLM caching services by applying provenance-aware key management.
Authors
Wei Song, Yuxin Cao, Xi Zheng, Leo Zhang, Xiao Cheng
Abstract
Production LLM serving stacks combine an inference engine's local prefix cache with a shared KV-cache tier for fleet-wide reuse. The local cache distinguishes requests by adapter, weight configuration and sharing domain, but the shared tier may key entries only by token content and coarse model metadata. This boundary erases provenance and lets identical tokens under incompatible computational or sharing contexts collide. We call this composition gap provenance-blind reuse and present its first systematic study. A source audit of three vLLM connectors confirms the structural omission, while runtime experiments reproduce it across vLLM and two SGLang releases, 12 models from 7 families (0.5 B-32 B), and over 160 configurations. Cross-adapter collisions reduce accuracy from 0.94 to 0.64, incompatible KV representations reduce reasoning accuracy to zero, and salt omission enables 93% prompt identification from timing. We formalize the missing guarantee as the KV provenance contract: for a declared dimension registry, shared keys must be injective over computational and sharing provenance, with identities stable across workers. Any dimension with a stable identity can therefore be added without connector-specific key logic. A canonical descriptor binds per-request and per-worker provenance into lookup and store keys, while a differential checker detects dimensions that change KV state without changing the key. Implemented in vLLM and SGLang 0.5.20 across three cache paths, provenance binding eliminates unsafe reuse while preserving legitimate sharing. Hit-path latency changes remain within 0.34 ms and below run-to-run variation; retention grows with provenance diversity.