Intent aware caching improves large language model conversations speed
Intent-Aware Caching for Efficient LLM Serving
Artificial IntelligenceMachine Learning
Summary
When people chat with large language models in multiple turns, computers try to save time by remembering parts of the conversation. But it's hard to guess which parts will be reused because people might stop or pause for different reasons. The authors found that knowing what the user intends to do helps predict if and when the conversation continues. They designed a new method that uses this idea to decide what conversation parts to keep in memory. This method made the model respond faster and reused conversation parts more often than older methods.
What this means in practice
- •For llm service operators: Reduce latency and computational cost in multi-turn conversational AI services by predicting conversation continuation with user intent.
- •For chatbot developers: Improve user experience with faster reply times by adopting intent-aware caching strategies for large language model chatbots.
Authors
Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos, Hans Vandierendonck, Deepu John, Bo Ji
Abstract
Modern LLM serving systems use prefix caching to accelerate multi-turn conversational workloads. They cache the key-value states of previous conversation prefixes and reuse them when subsequent requests share the same prefix, to avoid redundant prefill computation and reduce serving latency. However, achieving a high cache hit rate remains challenging because cache reuse depends on whether a conversation continues and how soon the next turn arrives. Our analysis of real workloads shows that user intent is a useful signal for this behavior, as different intents exhibit different continuation probabilities and inter-turn delays. Based on this key insight, we propose Intent-Aware Caching (IAC), an online eviction policy that classifies requests by intent, learns lightweight reuse statistics from recent serving logs and leverages these statistics to guide an oracle-inspired eviction strategy. Extensive evaluation on real workloads (WildChat and ShareChat) shows that IAC reaches a 34.6% cache hit rate. This substantially outperforms both the previous learning-based approach (18.8%) and traditional methods such as least recently used (LRU, 12.2%), while reducing average serving latency by up to 46.1%.