C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
2026-07-20 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study how to make large language models work faster with very long inputs, by reusing and compressing stored information (key-value caches) during inference. They recognize that saving storage and access time is as important as cutting computation, which past methods missed. Their new approach, C²KV, learns a special compressed and flexible way to store these caches so they can be reused easily without changing the original model. Tests show that this method greatly speeds up inference and reduces storage needs without hurting output quality.
large language modelslong-context inferencekey-value cachecache compressionreuse methodsinference speedupstructured attentioncompression tokenspretrained models
Authors
Chuheng Du, Junyi Chen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Chaoyue Niu, Shengzhong Liu, Guihai Chen, Fan Wu
Abstract
Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C$^2$KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C$^2$KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C$^2$KV significantly reduces KV cache storage and transfer costs, achieving up to 17$\times$ inference speedup under long contexts, while preserving generation quality.