FlashBack improves memory use in continuous video vision language models
FlashBack: Knowing When to Remember in Streaming Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Streaming vision-language models watch videos continuously but have limited computing power. The authors propose FlashBack to help these models decide when to use past information and when to rely on current input. FlashBack selectively retrieves relevant history without interrupting the current video analysis. This approach improves performance on tasks needing long-term memory without extra training. The method works with existing models and keeps up with real-time video understanding.
What this means in practice
- •For video analytics teams: Enhance continuous video analysis systems by selectively incorporating relevant past frames for improved long-term understanding without extra training.
- •For security monitoring providers: Improve real-time surveillance systems to recall important historical context on demand, enhancing threat detection and event interpretation.$Commercial implications: Enables more effective, real-time surveillance products that adjust memory use dynamically, making systems smarter and more responsive.
Authors
Yi Chen, MingMing Yu, Rui-Qi Wang, Boran Wang, Xiaohang Cao, Chu Tang, Jingmin Chen, Jie Gu
Abstract
Streaming vision-language models must process continuously growing video streams under a bounded compute budget, creating a persistent tension between real-time perception and long-term memory. Retrieving historical information provides a natural remedy, yet historical recall is not uniformly beneficial: unnecessary history may introduce irrelevant context into current reasoning and interfere with native real-time perception. Effective streaming memory should therefore address not only what to remember, but also when and how to access it. To this end, we introduce FlashBack, a training-free framework for selective, multi-level memory in streaming vision-language models. Before retrieving history, FlashBack draws on the semantic understanding of the frozen streaming VLM to infer whether a query calls for historical evidence. This assessment determines whether inference remains on the Native trajectory or invokes an isolated Recall trajectory. The Recall trajectory combines recent context with retrieved long-term memory through a query-local Side-KV pathway, preserving local temporal continuity without modifying the persistent Native state. We instantiate FlashBack on StreamingVLM and Mage-VL-4B and evaluate it on OVO-Bench and StreamingBench. The results show improvements on several long-horizon and memory-dependent tasks while largely preserving real-time perception, with performance competitive with strong training-based streaming methods despite requiring no additional training. Our code will be announced later.