Kv cache reuse varies across language model updates for better efficiency
Can KV Cache Be Reused Across Model Updates? A DeepSeek-V4-Flash Case Study
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
When language models get updated, they often have to recompute information they've seen before, which takes time. The authors studied whether reusing the stored memory of past computations, called the key-value cache, works across different versions of a model. They found that the cache can still help speed up answering questions, even if it doesn't exactly match the new model's behavior. However, the success depends on the task and how long the memory is, with some tasks losing accuracy.
What this means in practice
- •For machine learning engineers: Cut inference time for updated language models by reusing computation caches when handling longer text contexts in question answering tasks.
- •For ai system operators: Optimize runtime efficiency by reusing cached intermediate states across model versions to speed up API responses in natural text generation scenarios.
Authors
Zhikun Tao, Fangyong Jin, Junping Zhao
Abstract
Reusing key--value (KV) caches across language model updates could avoid repeated computation of shared context, but changes in model weights raise a fundamental question about what these caches preserve. We investigate direct cache reuse across four model updates, with a detailed study of DeepSeek-V4-Flash to Flash-0731. Our central finding is that caches can retain useful task information without faithfully reproducing the updated model's behavior. Retrieval accuracy varies substantially across updates, ranging from near-native performance to large losses. With source caches already available, reuse achieves median time-to-first-token speedups of $2.10$--$5.08\times$ on three longer-prefix QA benchmarks. To characterize the behavioral differences behind these outcomes, we introduce Functional Cache Divergence, which compares next-token distributions under reused and native caches. Direct KV cache reuse better preserves native model behavior in natural-text continuation, whereas compatibility degrades with increasing prefix length in single-token retrieval tasks. Answer-margin analyses help explain how correct retrieval decisions survive substantial internal state changes. Yet even agreement on the first generated token does not ensure agreement in answer content or stopping behavior. These findings establish task utility and fidelity to native inference as distinct criteria for evaluating cross-version KV cache reuse.