Query-adaptive reading improves key-value cache efficiency and accuracy
Read What Matters: Query-Adaptive Quantization for KV Caches
Machine LearningComputation and LanguagePerformance
Summary
Key-value caches store information ahead of time, but different questions (queries) need different parts of that information for accurate answers. The authors propose a method where each query decides which parts of the stored data it reads, allowing efficient and precise access without changing the storage. This method reduces the amount of information read per query while keeping answers accurate, which helps speed up long-context language model tasks. They tested this approach on multiple models and found it reduces computing time and preserves accuracy compared to traditional methods.
What this means in practice
- •For language model engineers: Reduce memory reads and speed up long-context decoding by adapting cache reading to each query's needs.
- •For cloud ai service providers: Lower latency for language model inference by using query-adaptive cache access, saving compute resources in deployed systems.$Commercial implications: Enables faster and more resource-efficient AI services with improved cache reading strategies.
Authors
Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
Abstract
KV-cache entries are stored before their future queries are known, but each decoding query needs precision in different places. We study this mismatch using separate budgets for retained bits and bits fetched per query. ReadKV stores each key and value in a progressive code whose prefixes support different reconstruction precisions. For each query, it allocates key-channel prefixes using the query, computes attention from the reconstructed keys, and then allocates value-token prefixes using that attention. Stored entries remain unchanged. Each stage optimizes a calibrated distortion objective under a fixed budget; we prove exact allocation under diminishing refinement gains and relate these objectives to attention-output error. We also exhibit a finite-dimensional attention family where query-dependent access strictly outperforms every query-independent reader at the same read budget, even with unrestricted competing encoders and decoders. Across six base models, reading four bits on average from an eight-bit cache increases C4 perplexity by at most 0.66%, using about one quarter of the logical reads and half the retained capacity of a 16-bit cache. It is consistently more accurate than storing and fully reading four bits at the same payload-read budget. Retaining more bits than each query fetches is aimed at long-context decoding, where the cache bytes moved per step, rather than the weights, dominate cost. Long-context question answering and retrieval on two instruction-tuned models provide additional quality evidence. On the tested 8K-token, batch-one, single-layer workload on an NVIDIA A10G, a restricted eight-bit ReadKV reader with a two-bit mean payload-read budget has 39% lower latency than the tested TurboQuant codec.