AgentVidBench tests AI models on complex video question answering
AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding videos like people do requires AI to think through multiple steps and reason about space, time, and cause-and-effect. The authors noticed that existing tests for video understanding were too simple and only asked one-step questions. To fix this, they created AgentVidBench, a new set of video questions that require AI models to reason through several parts and explain their thinking. Their experiments show current AI models struggle with these complex tasks unless they use special step-by-step reasoning methods.
What this means in practice
- •For ai developers: Evaluate how well AI models can reason over multiple steps in videos using a benchmark that includes detailed reasoning traces.
- •For robotics engineers: Test video understanding systems that require complex spatial and temporal reasoning tasks with a challenging multi-hop question set.
Authors
Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee
Abstract
Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent advances in Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in video understanding, existing benchmarks remain confined to simple scene-level queries or global summaries that require only single-step inference. Real-world video understanding involves more challenging tasks that require multi-hop multimodal reasoning, and there is a critical absence of video benchmarks equipped to rigorously evaluate these agentic capabilities. To bridge this gap, we introduce AgentVidBench, a multi-hop video question answering benchmark focused on evaluating the spatial, temporal, and causal reasoning capabilities of MLLM agents. Beyond standard question-answer pairs, AgentVidBench provides step-by-step solution traces to support trajectory evaluation that assesses whether agents explicitly acquire the evidence needed to justify their answers. Experiments with 12 proprietary and open-source MLLMs show that single-turn performance remains limited on AgentVidBench, while integrating these models into state-of-the-art agentic workflows generally improves performance with respect to both accuracy and trajectory scores. We further present a simple yet effective agentic strategy that serves as a competitive baseline on AgentVidBench, establishing our benchmark as a holistic testbed for future research on agentic video understanding. Code and datasets are available at https://github.com/krafton-ai/agentvidbench and https://huggingface.co/datasets/agentvidbench/agentvidbench.