Papers for

web service engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Semantic caches improve response reuse while controlling errors in LLM services

Semantic Caches for LLM Web Services Leave Their Error Budget Unspent

Abstract: Web services built on large language models (LLMs) receive many paraphrases of earlier queries, which a semantic cache answers from stored responses. A reused answer can be wrong, so wrong answers must stay below a share $δ$ of the queries. Existing caches reuse above a fixed similarity threshold or, like vCache, learn a per-entry rule that bounds each query's error by $δ$. However, fixed thresholds break the budget on new streams, and per-entry rules need labels on every entry before reusing it. We find an unspent error budget, as vCache realizes only 12-55% of $δ$ on three public query streams. Surplus spends this budget where reuse is most likely correct. It ranks queries by a risk score shared across paraphrase clusters and opens risk bins as far as an audited error estimate allows. Free labels from every LLM call and a ledger of realized errors keep this threshold safe under drift. On five streams, including ORCAS search queries, Surplus holds $δ$ in every test cell, which no baseline does. On the shuffled streams it serves 40% more hits than vCache, and its pooled hit rate beats the best baseline holding $δ$ in each cell by 4.10 points. In a vLLM testbed it cuts the mean latency by 31.59% at the highest load, against 23.67% for vCache.

Mon 2 NovMachine Learning
The gist
Large language model web services get lots of similar questions that can be answered using stored previous responses, called semantic caches. However, reusing old answers can sometimes be wrong and must be limited. The authors found that current systems are too cautious and don't use their full allowance of errors, missing chances to answer more queries from cache. They propose a method called Surplus that ranks query risks, carefully controls reuse error rates, and adapts over time, improving accuracy and speed. Testing shows Surplus serves more cached answers without exceeding error limits, reducing latency significantly.
Open → 2611.02498v1