KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors introduce KVDiagnosis, a dataset and benchmark for better understanding how KV-cache compression impacts long-memory in language models. They create a detailed system to categorize different compression methods and carefully analyze when and why correct results fail after compression. Using various tests, they identify cases where compression changes performance and show that boosting certain evidence-related attention can fix some failures. Their work helps clarify the weaknesses of compressed caches and provides tools to improve them.
KV-cache compressionlong-context memorylanguage modelsdiagnostic datasetattention mechanismslikelihood driftAUROCQwen3-8Bevidence-aware workloadscache compression evaluation
Authors
Chen Qiu, Ziwu Liu, Chao Fei, Guozhong Li, Panos Kalnis
Abstract
KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-method taxonomy groups methods into five mechanism families and links them to eight verified implementations and their valid diagnostic measurements. Second, for every supported method setting, we evaluate all sources in each fixed split against a per-source FullCache control before selecting FullCache-correct/compressed-wrong (C-to-W) rows separately for each method-setting, so no compressor defines another's test set. Third, a common record format links paired outputs and run metadata to cache, likelihood, attention, and decoding measurements with explicit applicability states. On Qwen3-8B, four evidence-aware workloads yield 59 800 supported compressed runs over 2600 sources and 12 520 C-to-W rows. Under fixed diagnostic rules, 63.2% have low or partial measured/projected coverage. Only 19 rows (0.2%) combine high measured/projected coverage with strong likelihood drift; another 2,126 (17.0%) preserve structural position addressability, for which representation fidelity remains unknown, while showing the same drift. Against C-to-C success controls, all ten diagnostics separate failed from successful compression (stratified AUROC 0.684-0.871). Among 96 reproducible low-EAR failures, a controlled 4x evidence-attention boost repairs 29.2%, versus 6.3% under a count-matched sham intervention and 3.3% degradation on matched C-to-C controls. Code and data are available at https://github.com/ChosenQC/KVDiagnosis.