Audio visual model improves reasoning on long videos with evidence
Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?
Computer Vision and Pattern Recognition
Summary
Understanding stories in long videos is hard because models get overwhelmed by lots of images and often ignore sound. The authors made a new test called Video-HolmesV2 that checks if models can use both sight and sound to answer questions and explain their answers with exact proof. They also developed a way to reduce confusing information by focusing on important parts of the video and audio. Even top models have trouble passing this test, but the new method the authors introduce works better than similar open models.
What this means in practice
- •For video analytics developers: Build video analysis tools that justify interpretations by referencing precise audio-visual moments in long footage.
- •For security monitoring teams: Implement systems that detect and explain events in lengthy surveillance videos using synchronized audio and visual cues.
- •For captioning service providers: Enhance automated captioning by integrating audio and visual signals to improve understanding over extended video narratives.$Commercial implications: Enables selling advanced captioning products that provide more accurate and context-aware transcripts for media companies.
Authors
Zhaoyang Wei, Zipeng Wang, Yushe Cao, Chenhui Qiang, Shuaibing Cheng, Xuesong Yang, Sen Nie, Bowen Jiang, Wenchao Ding, Yanchao Hao, Zheng Wei, Xuehui Yu, Zhenjun Han
Abstract
Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to "silent observers" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.