Safe-kv improves document reuse in ai text generation systems
SAFE-KV: Causal Suffix Repair for Behavior-Preserving Non-Prefix KV Cache Reuse in RAG
Computation and LanguageMachine Learning
Summary
When AI systems generate text using information retrieved from documents, they often reuse pieces of previous calculations to save time. However, these reused pieces depend on the order and context of earlier text, so simply reusing them without adjustments can cause mistakes. The authors describe a new method called SAFE-KV that carefully repairs these reused pieces by re-encoding parts of the context, ensuring accurate results even when reusing non-starting segments. Their approach restores nearly perfect accuracy in tests, making document reuse more reliable.
What this means in practice
- •For language model engineers: Implement SAFE-KV to enable accurate reuse of partial document states in retrieval-augmented generation, improving efficiency without loss of response quality.
- •For ai system architects: Design caching mechanisms for large transformer models that include causal suffix repair to maintain consistent outputs across requests reusing document context.
Authors
Xingjian Zhang
Abstract
Retrieval-augmented generation (RAG) reuses the same documents across many requests, making non-prefix KV-cache reuse attractive. Yet a cached document state is not reusable text: it is a context-conditioned causal state whose keys and values depend on the prefix, position, and preceding documents under which it was computed. We formulate non-prefix KV reuse as a behavior-preservation problem with full prefill on the exact prompt as the reference, and introduce SAFE-KV, which repairs a composed cache by re-encoding the causal suffix from a selected start document. On Qwen2.5-7B-Instruct, prefix-conditioned composition preserves only 47.2% of HotpotQA-500 and 37.8% of 2Wiki-500 full-prefill answers; SAFE-KV with risk-guided K2 selection restores 99.8%/98.4% (donor-copy protocol) and 100.0%/98.4% with actual suffix forwarding. Document-only and fixed-window repairs recover far less, because the document states after the repaired region keep histories that omit it. Reuse without repair is exact up to four documents and degrades at sixteen, where suffix repair recovers 98-100%. Faithful repair re-encodes most of the context, so in a single-request L40S study SAFE-KV's operating rule serves every workload by full prefill. SAFE-KV thus establishes the causal-state contract for non-prefix reuse and a validated repair mechanism that meets it.