Key value cache eviction improves large language model memory use
Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
Computation and Language
Summary
Long conversations with large language models need smart ways to remember important information. The authors studied how to decide what to forget by looking closely at the math behind how these models pay attention to words. They created a new method called Jacap that better chooses which information to keep or remove, making the model perform better, especially when memory is tight. This method helps the model focus on the most useful parts of the conversation for better results.
key-value cachelarge language modelscache evictionsoftmax attentionJacobianinformation capacityTaylor expansionsubset selectionstatistical leverage scores
Authors
Jiaming Yang, Chenwei Tang, Liangli Zhen, Chenyang Zhang, Jiancheng Lv
Abstract
Key-value (KV) cache eviction is essential for scaling long-context inference in Large Language Models. However, existing policies predominantly rely on empirical heuristics, lacking a rigorous characterization of token utility under the inherently nonlinear softmax attention mechanism. In this work, we rethink KV cache eviction through the lens of local information geometry, modeling the attention process as a nonlinear Gaussian communication channel. By performing a first-order Taylor expansion of the attention mapping, we derive the Jacobian Information Capacity, a novel objective that explicitly captures query relevance, softmax sensitivity, and structural diversity. Guided by this theory, we introduce Jacap, a capacity-aware eviction method that utilizes softmax-aware importance weighting and statistical leverage scores for subset selection. Extensive experiments across diverse architectures and benchmarks demonstrate that \textsc{Jacap} delivers superior performance in most scenarios, particularly in high-compression regimes.