Visual cache reuse cuts latency in vision language model inference
Visual Cache Reuse for Efficient Multimodal Inference
Operating Systems
Summary
Vision-language models take a long time and lots of memory to process big images combined with text, partly because they handle many visual tokens. The authors introduce VisMemo, a system that smartly reuses parts of previous computations to save time and memory. It uses a clever method to decide which visual information to reuse approximately without harming the output quality. Tests show VisMemo nearly doubles cache hit rates and cuts processing delays considerably on large inputs.
What this means in practice
- •For cloud service providers: Speed up multimodal AI services by reducing latency and memory use in large-scale vision-language model deployments.
- •For video analysis teams: Improve responsiveness of systems that analyze video frames with language queries by making cache reuse more effective even with approximate matches.
Authors
Shuheng Gao, Yuankai Fan, Jiaqiang Ma, Jiawei Shao, Yongkun Li, Xuelong Li
Abstract
Vision-language model (VLM) inference incurs substantial prefill latency and memory overhead due to the long visual token sequences of large visual inputs. Existing serving systems typically exploit exact cache reuse through prefix caching or position-independent caching, overlooking the potential of approximate caching in real-world workloads. However, directly reusing an approximate cache entry may introduce spurious visual information, which naive approaches that select a subset of important visual tokens for recomputation fail to adequately mitigate this risk. We present VisMemo, an efficient serving system that enables approximate KV cache reuse for modern VLMs. Its core is a system-algorithm co-design that combines an entropy-aware KV cache reuse algorithm with differentiated cache management. Specifically, VisMemo uses patch entropy to identify reuse candidates and warmup attention to prioritize recomputation based on the current query. To support the large cache capacity across GPU memory, CPU memory, and SSD storage, VisMemo applies distinct I/O and cache policies to exact and approximate hits: sparse patch-level I/O for approximate hits, prioritized retention for exact hits, and a reserved CPU victim cache for preset entries selected using offline workload priors and online reuse observations. VisMemo further optimizes the memory access path between the inference engine and the storage backend, eliminating redundant caching and unnecessary data movement. Experiments on multiple public datasets show that VisMemo preserves output quality under approximate reuse. On a real-world production trace, VisMemo increases the overall cache hit rate from 27.78% to 61.45% and reduces prefill latency by up to 48.3% at 8K input tokens.