Benchmark reveals overconfidence in video question answering systems
STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence
Computer Vision and Pattern Recognition
Summary
Answering questions about videos while the video is still playing is hard, especially when the system has to decide if there is enough information to answer or if it should say "I don't know yet." The authors created STORM-Bench, a dataset to test how well systems handle changing and missing information in videos. They found that current video question answering models often give answers even when uncertain, showing they are overconfident and less reliable than overall accuracy suggests. This benchmark helps highlight which models track information well and know when to abstain.
What this means in practice
- •For video analytics engineers: Improve video QA systems by testing their ability to handle uncertain or incomplete visual information during live video streams.
- •For surveillance system developers: Evaluate and reduce overconfidence in video-based event detection where evidence is partial or evolving over time.
Authors
Siru Zhong, Shenghan Tan, Rihong Yan, Xiaohui Lv, Yuzheng Zhuang, Shuai Tao, Wulong Liu, Haohuan Fu, Yuxuan Liang
Abstract
Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing benchmarks focus on static recognition or long-range retrieval, rarely evaluating these coupled capabilities under evolving and incomplete evidence. We present STORM-Bench, comprising 5,736 questions across 630 compact, change-dense episodes spanning five egocentric domains (STORM-Real) and two controlled simulation subsets (STORM-Sim) at 1 FPS. Questions are stratified by a proxy for accumulated change intensity (Low, Medium, High) and query-time answerability (Known, Uncertain). To measure reliability, we introduce STORM-BR, a harmonic metric over joint answer-status correctness that exposes abstention failures masked by aggregate accuracy, alongside STORM-BR-ATTR for uncertainty attribution. Across 14 video LLMs, online accuracy peaks at 60.3\% (mean 51.7\%), whereas STORM-BR ranges from 5.7\% to 35.6\% (mean 18.8\%), driven by pervasive overconfidence on uncertain queries. STORM-Bench shows that task accuracy masks these gaps in epistemic reliability and state tracking. Benchmark and code are available at https://github.com/siruzhong/STORM-Bench.