Benchmark tests AI agents by having them rebuild real videos in Blender
BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender
Computer Vision and Pattern Recognition
Summary
If a computer truly understands a video, it should be able to recreate it exactly using a program. The authors created a test called BVB where AI agents try to rebuild real-world videos as 3D scenes in Blender software. They then measure both how similar the recreated video looks and how many factual details are preserved. Their tests show that while AI can get close visually, keeping all the facts right remains difficult.
What this means in practice
- •For software developers: Use BVB as an objective way to test how well AI agents understand video content by making them reconstruct videos accurately.
- •For computer vision teams: Improve video analysis tools by identifying gaps in AI’s ability to retain factual video details during programmatic reconstruction.
Authors
Yolo Y. Tang, Daiki Shimada, Jiayue Meng, Jing Bi, Pinxin Liu, Yicheng Wang, Yunzhong Xiao, Zhangyun Tan, Zeliang Zhang, Chao Huang, Susan Liang, Qianxiang Shen, Luchuan Song, Ali Vosoughi, Mingqian Feng, Melika Filvantorkaman, Chenliang Xu
Abstract
Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmarks still evaluate models mainly through question answering. If an agent truly understands a video, it can reconstruct it programmatically. We introduce BVB, Blender-VideoBench, a benchmark that tests this ability by asking agents to reconstruct real-world videos as animated Blender scenes. To ensure fair comparison, each agent programs the reconstruction through a lightweight harness, Mini-BVB, in an identical sandbox under a shared cost limit. The benchmark renders each reconstruction from its animated camera and evaluates it on two axes: (1) Dual VQA measures how many spatiotemporal facts the reconstruction preserves. (2) Latent Similarity measures how closely the reconstruction matches the source video perceptually. Our overall score, a square-root mean, favors balanced performance. We evaluate 51 configurations from 10 model families and analyze semantic retention, perceptual similarity, reasoning effort, and cost. The best model reaches 88.6 Latent Similarity but retains only 53.7% of the source-correct spatiotemporal answers. Additional reasoning improves visual similarity but does not close this gap in factual accuracy. In a blind study with 15 raters and five configurations, Latent Similarity correlates strongly with human preference. These results show that programmatic reconstruction is a viable test of agentic video understanding, and that semantic retention remains the main challenge.