Dynamic Hub-and-Spoke Memory for Streaming Video Understanding
2026-08-31 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address the problem of understanding videos that keep growing over time, where it's hard to remember and use all past information. They propose a method called Dynamic Hub-and-Spoke Memory (D-HSM) that stores older video parts as text-based summaries connected by key entities, while keeping recent frames as images for detailed viewing. When a question is asked, their system picks out relevant past information and combines it with current visuals to answer accurately. Their experiments show this approach improves performance compared to other methods both for live and recorded videos.
streaming video understandinglong-range historytextual memoryvisual tokenshub-and-spoke memoryentity-centered memoryfrozen-VLMquestion-aware retrievalvideo benchmarksonline video understanding
Authors
Xinru Jiang, Lin Zhao, Xi Xiao, Yunbei Zhang, Janet Wang, Chenrui Ma, Haolin Li, Yanzhi Wang, Yifan Gong, Octavia Camps
Abstract
Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.