DeltaStreamer improves video understanding by remembering changes and replaying details
DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding
Computer Vision and Pattern RecognitionArtificial IntelligenceHuman-Computer Interaction
Summary
Understanding videos as they play is tricky because you don’t know what questions will come later. The authors built DeltaStreamer to watch videos like humans do: by noticing what changes, remembering events, and then revisiting parts of the video when asked a question. This helps the system not miss important details and answer questions better. Tests show DeltaStreamer does better than other methods on video understanding challenges.
What this means in practice
- •For video analytics teams: Provide tools to analyze live video streams with better detail retention and query-based evidence retrieval to answer event-related questions accurately.
- •For security monitoring operators: Enhance continuous surveillance systems by enabling focused reviews of past footage triggered by specific incident queries to recover overlooked visual evidence.
Authors
Boyuan Sun, Deshui Miao, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Chi Liu, Yan Shu, Derek Li, Bryan Dai, Shengsheng Qian, Qibin Hou
Abstract
We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/