When to evict cache entries improves large language model performance
When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
Machine LearningComputation and Language
Summary
Large language models remember previous words in a special cache to help generate better answers. Some methods try to compress this cache by deciding which words to keep before starting to answer. The authors found this timing can hurt real quality because the model hasn’t yet started forming answers when those decisions are made. They propose waiting just a tiny bit longer—after the model drafts a couple of answer words—before deciding which cache entries to remove. This simple timing change improves performance across several tested models without extra training or complex changes.
What this means in practice
- •For machine learning engineers: Improve inference efficiency for large language models by applying draft-guided eviction to better manage cache memory during token generation.
- •For cloud ai service providers: Deploy draft-guided eviction to reduce memory usage and maintain answer quality in large language model APIs without retraining models.$Commercial implications: Enables more cost-effective large language model services by optimizing cache compression with only timing changes, improving performance at scale.
Authors
Haeyong Kang, Chang D. Yoo
Abstract
Training-free KV-cache compression methods such as SnapKV, H2O, and PyramidKV evict tokens at the end of prefill, aiming to preserve the attention mass that future queries are expected to use -optimizing what to keep. We show that this objective fails in two distinct ways. (1) Compensation: restoring the evicted attention mass can recover the attention-level target without recovering task quality. (2) Selection: covering more of the true decode-query mass can hurt quality when the recovered mass is fragmented rather than concentrated in coherent spans. These failures share a common cause: eviction occurs before the queries that determine the answer trajectory exist. We propose Draft-Guided Eviction (DGE), which defers eviction until after drafting the first k=2 answer tokens using the full cache - just one decode step beyond prefill. Because the draft is generated from the answer's own prefix, no cache entries are discarded before this trajectory signal becomes available. The per-head cache budget remains unchanged, and DGE can be applied directly to SnapKV, PyramidKV, H2O, and StreamingLLM without modifying their eviction scores. Unlike extra-pass methods, DGE changes when eviction occurs rather than what cache entries are selected. Extensive experiments demonstrate that DGE outperforms prior methods at every evaluated budget on five of six instruct-tuned backbones, achieving 44.2 on LongBench, nearly matching FullKV at 44.3. The timing-only control DGE-W achieves the same score, demonstrating that the gain comes from when eviction occurs rather than what is selected - an effect we term trajectory anchoring.