Video large language models struggle to verify detailed event descriptions
VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding exactly what happens in videos is hard for AI systems called Video Large Language Models (Video-LLMs). The authors created a new test called VidOmni-Bench, with 500 videos and detailed descriptions, to check if these models can tell which events really happen in the videos. They found that these models often make mistakes by describing things that aren’t there and have trouble spotting wrong details. The study also shows that errors vary depending on how complicated and long the videos are.
What this means in practice
- •For video platform developers: Improve quality control by detecting incorrect events in automated video captions to enhance user experience on platforms with diverse video content.
- •For media monitoring teams: Verify the accuracy of automatically generated video descriptions to ensure reliable content indexing and retrieval from videos of varying length and complexity.
Authors
Changbeen Kim, Junwon Chang, Kipyo Kim, Risa Shinoda, Kuniaki Saito, Donghyun Kim
Abstract
While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understanding remains challenging. Existing benchmarks often rely on question answering or ground-truth caption matching, where models may succeed through superficial cues and incomplete annotations. To this end, we introduce VidOmni-Bench, a benchmark that requires models to verify whether each event in dense video captions is supported by the video. VidOmni-Bench consists of 500 videos spanning five complexity types and diverse durations from 4 seconds to 90 minutes. After collecting videos along these axes, we use diverse Video-LLMs to generate dense captions and obtain human-verified sentence-level labels, where sentences containing incorrect events serve as hard negatives for evaluation. Our experiments on VidOmni-Bench reveal three key findings: (i) Video-LLMs frequently generate hallucinated descriptions in dense video captioning; (ii) they also struggle as verifiers, failing to reliably detect plausible but incorrect event descriptions; and (iii) model weaknesses vary across video complexity and duration, revealing diverse, model-specific bottlenecks in current Video-LLMs.