Streamclarify improves accuracy in real-time video question answering
StreamClarify: Proactive Clarification for Grounded Streaming Video Question Answering
Computer Vision and Pattern Recognition
Summary
Answering questions about streaming videos quickly is tricky because computers can misunderstand ambiguous questions or forget important parts of the video. This paper presents StreamClarify, a system that helps decide when to answer a question immediately and when to ask for clarification. It uses measures of uncertainty and keeps a memory of recent and past video frames to improve answers without slowing down. The authors show that StreamClarify makes video question answering more accurate, especially when questions are unclear or unanswerable.
What this means in practice
- •For video analytics developers: Improve real-time video question answering systems by integrating uncertainty-based clarification to handle ambiguous or missing visual context.
- •For customer support teams: Enable interactive video assistants that proactively request clarification to improve answer accuracy in continuous video streams during support calls.$Commercial implications: Makes possible commercial interactive video support agents that reduce errors in automated answers by clarifying user intent.
Authors
Zhuchen Shao, Bo Chen, Klara Nahrstedt
Abstract
Streaming video question answering (QA) requires real-time reasoning over continuous video streams under strict latency and active-context budgets. Current streaming vision-language models (VLMs) often give confident yet ungrounded answers when queries are ambiguous or have false premises, or when relevant visual evidence is no longer in the active context. Proactive clarification can help, but must satisfy the same latency and memory constraints. We present StreamClarify, a training-free framework for grounded streaming QA that combines a pre-answer uncertainty check with hierarchical memory. The check uses the entropy of a binary pre-answer judgment as a lightweight uncertainty score to decide whether to answer directly or request clarification. To support re-answering, a temporal-pyramid KV cache maintains a bounded active visual context, while a CPU-side semantic retrieval archive provides access to historical frames. We further construct StreamClarify-QA, a diagnostic benchmark of 1,837 QA pairs from StreamBench and OVO-Bench, spanning standard, referential ambiguity, and unanswerable scenarios. Across three VLM backbones, StreamClarify improves Macro accuracy by up to 8.2 points over backbone-matched baselines. At comparable trigger rates, entropy-based selection achieves more than 18 points higher precision than random selection in identifying initially incorrect answers. Under oracle user guidance, clarification improves Macro accuracy by up to 3.5 points. We also report a controlled, oracle-assisted Qwen3-VL simulation on Ambiguous QA with generated questions and referent-only user replies, where the entropy-gated condition improves accuracy by 8.3 points.