SPARE improves streaming video question answering with adaptive evidence retrieval
SPARE: Shared Perception and Adaptive Retrieval of Evidence for Streaming Video Understanding
Computer Vision and Pattern Recognition
Summary
Understanding video as it plays in real time can be tricky because sometimes you need to remember earlier parts, but too much old information can be confusing. The authors created SPARE, a method that decides when to look back at past video clips to help answer questions better, while focusing on recent important moments. It uses shared visual processing to stay efficient and learns to combine current and past information smartly. This approach helps answer questions about videos more accurately than some existing models.
What this means in practice
- •For video analytics teams: Enhance real-time video question answering systems by selectively retrieving past relevant clips to improve accuracy.
- •For surveillance system developers: Build smarter monitoring systems able to reference historical footage adaptively when interpreting ongoing video events.$Commercial implications: Enables commercial surveillance solutions to deliver improved real-time incident understanding using adaptive retrieval.
Authors
Senbin Xu, Zijun Zhang, Xinhua Wu, Li He, Jubo Feng, Li Qiao, Zhen Gao, Yan Zhang, Jixiang Luo, Rongrong Ji
Abstract
Streaming video understanding requires answering questions from the observed video, with evidence drawn from recent observations or earlier events. Historical context can improve recall but may interfere with judgments based on recent frames. Its usefulness depends on both when it is introduced and which clips are selected, while relevance alone does not guarantee sufficient evidence for an answer. We propose SPARE, a framework that combines adaptive historical retrieval with visual encoder sharing. A lightweight gate predicts whether recent observations provide sufficient evidence to answer the question. When additional evidence is predicted to be necessary, a multimodal embedding model retrieves relevant historical clips to supplement the current context. LoRA instruction tuning trains the understanding model to use recent observations and retrieved history jointly and abstain when the evidence remains insufficient. To reduce redundant visual parameters, the understanding and retrieval branches share a visual encoder while retaining separate language backbones. Four linear adapters align the shared features with the retrieval model through distillation at the visual token and final retrieval embedding levels. With visual-only inputs, SPARE achieves 77.78% on the evaluated Real-time and Backward task groups of OVO-Bench and 73.31% on StreamingBench. Compared with Qwen3-VL-8B-Instruct evaluated at 2 fps, SPARE improves accuracy by 20.68 percentage points on OVO-Bench and 14.15 percentage points on StreamingBench.