Kv repurpose enables flexible reuse of kv cache in large language models
KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
Computation and Language
Summary
Large language models often save parts of previous computations in a structure called a KV-cache, but usually they can only reuse this cache when the new input starts exactly the same as an old one. The authors created KV-Repurpose, a system that allows reuse of cached parts even when the inputs are not exact matches, like when the same document is asked about after different conversation starters. They show this can greatly increase cache reuse and speed up responses, especially under certain workloads, without much loss in answer quality. However, this method has limits when questions require connecting information across multiple documents.
What this means in practice
- •For ai service engineers: Improve response speed and efficiency in chatbot and language model services by reusing cached computations across varied prompts sharing document segments.
- •For cloud infrastructure teams: Optimize cache capacity usage for language model deployments by implementing KV-Repurpose’s segment-wise reuse to reduce token processing time.
Authors
Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
Abstract
KV-cache reuse in large language model serving is commonly limited to exact prompt prefixes, preventing reuse when the same retrieved document appears after different system prompts, conversation histories, or dynamic metadata. We present KV-Repurpose, an open-source, vLLM-based system for developing and evaluating segment-wise KV-cache reuse beyond exact prefixes. The system combines cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments on long-context benchmarks show that naive segment reuse largely preserves output quality for single-document question answering and summarization, while exposing clear limitations in cross-document reasoning. Under a Zipfian RAG-style workload in which 20% of requests share a fixed prefix, KV-Repurpose achieves a substantially higher token-level cache hit rate than prefix caching at the same 30 GB cache capacity, 63.9% versus 3.3%, resulting in lower TTFT. Even with 1 TB of cache capacity, prefix caching reaches only a 17.9% hit rate in this workload. Source code is available under the Apache-2.0 license at https://github.com/taiga333/eval-kv-repurpose.