Models learn when to ask questions during streaming video watching
AskStream: When and What to Ask Proactively under Uncertainty in Streaming Video Understanding
Computer Vision and Pattern RecognitionComputation and Language
Summary
Video understanding systems often wait passively or answer questions without checking if they have all the needed information. The authors of this paper found that uncertainty in videos or user queries can cause these systems to either wait indefinitely or guess wrongly. They propose that systems should have a third option: to proactively ask questions when uncertain before answering. To study this, they created AskStream, a new dataset and benchmark that tests when and what models should ask under uncertainty, showing current models still struggle to do this well.
What this means in practice
- •For video technology teams: Develop video analytics systems that decide when to ask for more details to improve real-time understanding accuracy.
- •For customer service chatbots: Enhance chatbots with the ability to ask clarifying questions during live video calls when information is incomplete.
Authors
Lu Wang, Zhuoran Jin, Longxiang Wang, Junhong Chen, Yupu Hao, Yao Xu, Daojian Zeng, Yubo Chen, Kang Liu, Guang Liu, Jun Zhao
Abstract
Recent advances in multimodal large language models have increased interest in interaction paradigms for video understanding. The evolution from offline to streaming and proactive interaction has gradually shifted models from passive responders toward more proactive agents that can decide when to answer. However, existing proactive paradigms typically assume that the information required to answer a query is complete, leaving models to only wait or answer. In the real world, however, information is often incomplete due to uncertainty from either the video or the user. In proactive paradigms, such uncertainty can cause the model to wait forever or generate highly hallucinatory responses. We therefore argue that proactive interaction should be redefined with a third action: ask, allowing models to actively ask for missing information. To study when and what models should ask before answering, we introduce AskStream, the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding, with four task categories and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows across three types: wait, ask, and answer. Across both multiple-choice and open-ended settings, even the strongest model attains only 52.18% on our proposed Ask-F1 metric, highlighting substantial limitations in proactive information seeking under uncertainty.