Arc-kv drastically speeds up long context cache compaction in large language models
ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction
Artificial Intelligence
Summary
Long input sentences slow down large AI language models because they keep lots of information stored for each word, which grows quickly and uses a lot of memory. The authors found a way to shrink this stored information faster by learning how to pick important clues (anchors) smartly, instead of searching for them many times. Their method, ARC-KV, makes a smaller memory footprint for long inputs that can be reused for many questions, speeding up processing while keeping accuracy high. Tests on well-known language benchmarks show ARC-KV is much faster and slightly more accurate than older methods.
What this means in practice
- •For ai infrastructure teams: Build more efficient long-context language model services by compressing KV caches faster without losing much accuracy.
- •For cloud ai service providers: Deploy language model APIs that handle longer conversations or documents at lower compute costs by reusing compacted KV caches.$Commercial implications: It enables offering faster, more cost-effective language model inference for enterprise customers requiring long context understanding.
Authors
Zheyu Shen, Guanhua Wang, Dezhan Tu, Mengchi Zhang, Yanjia Li, Adnan Aziz, Chunqiang Tang, Ang Li
Abstract
Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task performance with compact KV caches. However, iterative anchor search dominates the compaction cost of OMP-based Attention Matching. This motivates our selective amortization principle of learning a reusable anchor-selection policy across contexts while retaining context-specific reconstruction. In this work, we propose ARC-KV, a novel reconstruction-based KV cache compaction method that follows this principle. To this end, we first train a value-aware indexer to select real-key anchors in a single scoring pass. ARC-KV then applies convex-hull-constrained key merging and fits an attention-mass bias and compact values against the full cache. At inference time, ARC-KV builds the compact cache once per context using the frozen indexer and reuses it for all subsequent queries. Extensive experiments demonstrate that ARC-KV outperforms reported compaction methods in most settings across QuALITY, RULER, and LongBench on Llama-3.1-8B-Instruct. In particular, at 10% KV retention on QuALITY, ARC-KV improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time by a factor of 25.73, from 959.8 s to 37.3 s.