Prompt caching errors in large language model tools cause hidden costs and delays
Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents
Software EngineeringArtificial Intelligence
Summary
Many apps use large language models (LLMs) to answer questions, which can be slow and expensive. To speed things up and save money, these apps try to reuse previous answers through a process called prompt caching. But the authors found that many apps make mistakes when using prompt caching, and these errors often don't cause obvious problems, making them hard to notice. The authors created a tool named CACHELINT to detect these hidden errors, helping developers fix them and reduce costs.
What this means in practice
- •For llm service operators: Detect and fix prompt-caching mistakes in deployed systems to reduce costly and hidden inference overhead.
- •For software developers: Identify silent caching errors during development of open-source and commercial LLM-based agents to improve correctness and efficiency.
Authors
Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, Leo Zhang
Abstract
Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.