KVpop -- Key-Value Cache Compression with Predictive Online Pruning
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors identify that storing key-value (KV) cache during language model decoding uses a lot of memory, slowing things down as context grows. They propose KVpop, a learned method that decides which cache entries to keep or drop by predicting future importance, instead of using simple fixed rules. Their approach uses a special target based on future attention without heavy computations and delays decisions to better use upcoming context. Tests show KVpop reduces memory use significantly while keeping almost the same accuracy as using full attention, outperforming older methods. This means efficient memory use is possible without much loss in quality during decoding.
key-value cacheautoregressive decodingKV evictionfuture-attentionsupervised learningmemory compressionattention mechanismlanguage modelscache policymathematical reasoning datasets
Authors
Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl, David Stap, Thomas Schmied, Sebastian Böck, Günter Klambauer, Sepp Hochreiter
Abstract
Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, which poorly track future token utility and cause brittle eviction as relevance shifts. To address this, we introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision. The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps. We further introduce a delayed memory-based scorer that, uniquely among learned eviction methods, defers scoring for a fixed number of steps to exploit near-future context. On AIME and HMMT mathematical reasoning, KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines. Qwen3-8B shows even stronger results, reaching near-full teacher performance. These results show that supervising eviction with future-attention signals cuts memory costs while maintaining quality.