Multi-level entity-aware memory improves streaming video understanding
MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding
Computer Vision and Pattern Recognition
Summary
Understanding long streams of video is hard because it is difficult to remember detailed information over time. The authors found that current methods keep only broad summaries and lose details about specific things or people in the video. They designed MEMO, a system that separates video into meaningful parts and remembers both the overall scene and individual entities in detail. This lets MEMO pick out important visual information on demand, helping it better understand videos without extra training. Their tests show MEMO improves performance on challenging video tasks when combined with existing models.
What this means in practice
- •For video system developers: Enhance streaming video applications by integrating MEMO for better long-term and detailed visual understanding without retraining existing models.
- •For surveillance system engineers: Improve continuous video monitoring systems by selectively retrieving entity-level visual details, aiding accuracy in long-duration analysis.
Authors
Yinying Li, Yuqian Fu, Yulin Dai, Jingyu Gong, Tianwen Qian, Xiaoling Wang
Abstract
Streaming video understanding requires models to process unbounded visual streams while preserving rich visual semantics across vast temporal horizons, posing a fundamental challenge for memory modeling. Existing approaches primarily focus on increasing memory capacity, either by compressing historical information into fixed-size representations or by extending storage beyond GPU memory. However, these methods largely rely on global or coarse-grained representations, inevitably losing fine-grained visual information. In this work, we argue that streaming video memory should explicitly encode structured and semantically meaningful representations, particularly at the entity level. To this end, we propose MEMO, a novel framework that models streaming video through multi-level, entity-aware structured memory. MEMO performs multi-level perception to jointly capture global semantics, entity dynamics, and spatial structures, partitioning streaming video into semantically coherent chunks. Each chunk is organized into a structured memory, where lightweight global and entity-level representations serve as retrieval indices, while the corresponding high-resolution visual content is retained separately for on-demand access. At inference time, MEMO performs query-specific retrieval over the structured memory and selectively recalls relevant visual evidence for downstream reasoning. Notably, MEMO is training-free and plug-and-play with existing multimodal large language models. Extensive experiments on StreamingBench and OVO-Bench demonstrate that MEMO consistently improves multiple base models and achieves state-of-the-art performance.