Mvvbench tests video models on 3d and time reasoning
MVVBench: Benchmarking 4D Reasoning in Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding events from multiple cameras at once is hard because you have to track things moving across different views and times. The authors created MVVBench, a new kind of test that asks questions only answerable by looking at several camera views and moments together. They checked how well current vision-language models do on these tough questions and found common mistakes. They also explored methods to improve models without retraining and showed some promise using reinforcement learning. This work helps measure and improve how AI systems understand complex video scenes from many cameras over time.
What this means in practice
- •For security monitoring teams: Improve understanding of overlapping camera feeds to track people and events across different rooms and times.
- •For robotics developers: Enhance robot perception by integrating visual information from multiple cameras to better understand object movements in 3D space over time.
Authors
Hyungjin Chung, Byeongjun Park, Joonseok Lee, Hojun Kim, Jaeho Choi, Byung-Hoon Kim
Abstract
Multi-view video understanding requires integrating spatial and temporal evidence across multiple, often non-overlapping camera streams: tracking entities as they transition between viewpoints, aligning events across time, and reasoning about latent 4D continuity rather than any single visible frame. We introduce MVVBench, a benchmark for multi-view video reasoning built from real world multi camera datasets. Questions are curated to be monocular-ambiguous along both the view and the temporal axis: each question is unanswerable from any single view in the designated input set, and the majority are further unanswerable from any single moment. Each question becomes uniquely solvable only by jointly reasoning across views and across time. MVVBench spans diverse dynamic scenes and probes six capabilities: implicit/explicit attribute identification, implicit/explicit relative distance, relative camera pose, and compositional counting, with human-authored QA and rigorous verification. Beyond benchmarking, we provide an extensive analysis of when and why current vision language models succeed or fail, characterizing errors due to temporal mis-localization, cross-view identity breaks, and brittle multi-hop reasoning. We then study inference-time elicitation strategies that unlock latent multi-view competence---task-specific chain-of-thought scaffolds and structured cross-view evidence aggregation---yielding substantial gains without retraining. Finally, we present preliminary evidence that reinforcement learning with verifiable rewards can elicit some latent multi-view competence in the base model, pointing to training-time approaches as a promising direction for future work. Together, MVVBench offers a rigorous evaluation of 4D multi-view reasoning and a foundation for future progress toward reliable embodied perception.