CacheReforge cuts recomputation and keeps language model outputs accurate
CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters
Machine Learning
Summary
Large language models use special memory called KV caches to speed up processing long conversations or texts. When small updates or changes called adapters happen, these caches can get out of date, causing the model to give wrong answers. The authors created CacheReforge, a method that carefully chooses which parts of the cache to recompute and which to reuse. This approach keeps the model’s answers accurate while saving a lot of time and computing resources.
What this means in practice
- •For natural language processing engineers: Maintain accurate and efficient long-context processing in evolving language models by selectively recomputing cache layers based on adapter updates.
- •For cloud ai service providers: Reduce the time and cost of updating cached model states when deploying new adapter versions for interactive language model services.
Authors
Yuhang Cao, Yanzhou Mu, Chunrong Fang, Zhenyu Chen
Abstract
Large language models rely on KV caching to reduce repeated prefill computation in long context and interactive applications. As lightweight adapters evolve, cached states reflect earlier versions, so stale reuse distorts current model outputs, while complete affected suffix recomputation restores fidelity at substantial cost. We seek minimal recomputation that recovers current adapter behavior. Existing systems track token, context, or stable adapter identity, but neither represent caches from earlier adapter versions nor distinguish update propagation from the recomputation required for behavioral recovery. To address these gaps, we introduce CacheReforge, which represents stale KV caches as layerwise mixed-version objects. It combines per-layer adapter anchors, calibrated sensitivity, accumulated drift, and executable restart boundaries to select direct reuse, bounded recomputation, or complete affected-suffix recovery. We distinguish dependency depth from the functional recomputation horizon and use cumulative tail influence to characterize when bounded recovery preserves current-model behavior. We evaluate CacheReforge on Qwen2.5-1.5B and Qwen2.5-7B with continual LoRA updates, including 16K HotpotQA and 2WikiMQA workloads. CacheReforge reduces mean KL divergence by 92.4% relative to stale reuse, while recomputing only 5.44% of layers and reducing cache-maintenance time by 93.2% relative to fresh full prefill. These results show that version-aware recovery preserves model fidelity and most KV caching gains.