RelaxKV improves cache reuse in language model generation with sparse query attention
RelaxKV: Recomputation Guided by the Query with Sparse Context Attention for Efficient KV Cache Reuse
Artificial Intelligence
Summary
Generating text using large language models often needs to remember past words efficiently to save time and computation. The authors study a way to reuse stored information from earlier requests more flexibly, fixing the problem that prior methods missed some important interactions between words. They introduce RelaxKV, which smartly decides what parts of the stored information to recompute by focusing only on what the new question needs, reducing workload. This method improves performance across several language models, making text generation faster and more accurate in some tests.
What this means in practice
- •For machine learning engineers: Optimize transformer inference cost by reusing and selectively recomputing KV cache states for faster response with long input sequences.
- •For cloud ai service providers: Reduce computation during client query processing by improving cache reuse, supporting more efficient handling of large-context language models.
Authors
Ruoling Qi, Yirui Liu, Xuaner Wu, Yuxin Jin, Jian Chen, Jiayu Qin, Yin Chen, Jiawei Shao
Abstract
Cross-request KV caching reduces the prefill cost of Retrieval-Augmented Generation (RAG), but conventional prefix caching severely limits cache reuse across requests. Position-Independent Caching (PIC) removes this constraint by reusing independent chunks, but their KV states miss cross-chunk interactions. Existing methods selectively recompute token states to recover these missing interactions, but primarily allocate the recomputation budget to selecting which states to recompute, while fixing the recomputation context to the full causal prefix. We introduce RelaxKV, which formulates selective cache repair as a joint allocation problem over repair targets and recomputation context. Guided by the user query, RelaxKV identifies layer-specific repair targets and restricts their recomputation to a query-relevant context, reducing attention computation. Across four decoder models, RelaxKV at a 15% anchor ratio improves aggregate LongBench performance over ProphetKV on all models. On Qwen3-14B, RelaxKV provides a stronger quality-TTFT trade-off than ProphetKV across a 5%-30% anchor-ratio sweep, and achieves the best selective results on RULER-MV and LV-Eval at 16K and 32K context lengths. Controlled ablations further demonstrate the importance of recomputation context selection.