Video large language models improve speed and accuracy by choosing when to look back
When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs
Computer Vision and Pattern Recognition
Summary
Understanding videos as they play is hard because computers must decide what parts of the video to remember and use to answer questions quickly. The authors created WRWS, a way for video language models to smartly pick which past moments to keep in mind by guessing how uncertain they are about their choices. This method helps models focus more on recent or relevant parts depending on the situation, saving time and keeping answers accurate. Tests show WRWS works well across many video models and makes answering faster than previous methods.
What this means in practice
- •For video app developers: Design video apps that answer questions or provide descriptions in real time by efficiently selecting important past video frames before heavy processing.
- •For media content platform teams: Speed up content analysis pipelines with models that allocate computing budget by focusing on recent or semantically relevant video parts, lowering processing delays.
Authors
Xiang Hu, Jiazuo Yu, Lu Zhang, Yunzhi Zhuge, Huchuan Lu
Abstract
Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.