KV cache reuse accuracy needs better testing methods
Evaluating the accuracy of KV cache reuse techniques
Machine Learning
Summary
Reusing parts of memory called KV caches helps make AI responses faster, but it can also make them less accurate. The authors found that current ways to test these memory reuse methods often give a false impression of accuracy by missing real losses. They also noticed that existing test datasets don't properly show when these memory reuse issues happen. So, the authors created a clearer way to measure accuracy loss and a tool called Boxoffice to build challenging test datasets that better reveal when KV cache reuse causes problems.
What this means in practice
- •For ai system developers: Assess the true accuracy trade-offs when reusing KV caches to speed up AI response times in retrieval-based generation.
- •For benchmark designers: Use the Boxoffice tool to create datasets that thoroughly test KV cache reuse behavior under realistic and difficult conditions.
Authors
Samuel Cestola, Tianxiang Xia, Pengfei Zheng, Weiyan Zheng, Bo Wang, Yi Zhao, Diego Didona
Abstract
Position-independent KV cache reuse aims to reduce latency in retrieval-augmented generation by reusing chunk-level KV caches across prompts. We show that current evaluations of KV cache reuse techniques rely on measurements that fail to faithfully capture the loss of accuracy attributable to reuse, often artificially inflating the reported effectiveness. We also show that existing datasets do not exhibit the reuse dynamics needed to thoroughly evaluate such techniques. To address these issues, we propose an evaluation methodology that measures this accuracy loss without ambiguity and we introduce Boxoffice, a tool that programmatically generates evaluation datasets that exercise challenging KV cache reuse patterns.