KV Cache Offloading for Context-Intensive Tasks
2026-04-09 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how a method called KV-cache offloading, which helps large language models (LLMs) use memory more efficiently, performs on tasks that need lots of detailed information from the text. They created a new test called Text2JSON that checks if models can pull out structured data from long text. They found that current offloading methods reduce accuracy, especially on models like Llama 3 and Qwen 3. The authors discovered specific causes for this problem and suggested a simpler fix that improves accuracy. Their work shows the importance of carefully testing memory-saving techniques in LLMs on complex tasks.
KV-cacheoffloadinglong-context LLMsText2JSON benchmarkstructured knowledge extractionLlama 3Qwen 3low-rank projectionlandmarksinference latency
Authors
Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev, Vyacheslav Zhdanovskiy, Yegor Yershov
Abstract
With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.