Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how compressing stored information in retrieval-augmented generation systems affects the trustworthiness of the generated answers. They tested their model with two levels of compression and found that mild compression (INT8) keeps answers both accurate and faithful to the source. However, stronger compression (INT4) often makes answers less faithful, meaning the answers may no longer be properly supported by the original information, even if they appear correct. This issue is worse when the retrieval data is noisy or large. The authors suggest checking faithfulness carefully before using compressed caches in these systems.
retrieval-augmented generationkey-value cachequantizationcompressionfaithfulnessaccuracyhallucination detectionnatural language inferenceLLM evaluationnoisy retrieval
Authors
Atta Ul Asad, Ahsan Bilal, Muhammad Ali, Muhammad Haseeb, Dean F. Hougen
Abstract
Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.