Code based video agents rated across diverse real world tasks
VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
Computer Vision and Pattern Recognition
Summary
Video generation models often cannot create specific and detailed videos like exact formulas or synchronized narration. The authors present VibeVideoBench, a new test set with 239 video tasks in English and Chinese, covering topics like lessons, data stories, and music videos. They tested nine top AI agents that write code to generate videos with sound. Even the best agents fail about a quarter of the time, often because they run too slowly or don’t finish rendering. The evaluation combines automated checks with human-like judge panels to rate quality and task success.
What this means in practice
- •For multimedia software developers: Test and improve AI agents that automatically generate videos with code for educational, musical, or explanatory content.
- •For content production teams: Evaluate AI tools that produce synchronized sound and visuals from text instructions to aid video creation workflows.
Authors
Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Han Li, Wentao Zhang, Weiran Xu, Kongming Liang, Zhanyu Ma
Abstract
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.