Summary
Large language models remember information from past words using special memory called a key-value cache, but this memory gets too big and slow when the models are large. The authors found that this memory looks quite different across layers of the model, but the inputs to these layers are similar. Using this insight, they designed BellowsKV, a way to store one shared compressed memory for many layers without much loss in accuracy. This method compresses the data by representing it in a small shared space and then reconstructing the layers’ memory from it, making the model faster and more efficient without needing to retrain. They tested it on popular models and showed it works better than other similar compression methods.
What this means in practice
- •For ai infrastructure teams: Reduce GPU memory use during language model inference by compressing cross-layer attention caches with minimal accuracy loss.
- •For enterprise chatbot developers: Run large multi-turn dialogue agents more efficiently on existing hardware by using BellowsKV’s compressed key-value cache for faster and cheaper deployment.$Commercial implications: Enables cost-effective interactive AI services by reducing resource demands while maintaining response quality.
Abstract
Large language models (LLMs) now serve as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To avoid recomputation, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens. All three lose substantial accuracy at high compression ratios. Cross-layer methods share one compressed cache across adjacent layers to go beyond this limit. However, these methods also degrade sharply at high compression ratios. We find that the KV-cache representations of adjacent layers are quite different, whereas the input hidden states of adjacent layers are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in two other ways: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one shared eviction decision per block, because the layers rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, where the strongest cross-layer method retains at most 84% at 16x. Code: https://github.com/MorrisZJ/BellowsKV