VideoVIBE: A Video-Grounded Diagnostic Benchmark for One-Shot Interactive Website Generation
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created VideoVIBE, a new way to check how good AI-generated web apps are by using videos of humans interacting with them. This method looks closely at different kinds of mistakes like design issues or broken functions, rather than just scoring the final result. They also made V2Lens, a tool that uses both the videos and webpage code to better understand and fix these problems without needing extra training. Testing with multiple models showed that combining video and code analysis gives a clearer and more detailed picture of app quality.
vibe codingvideo-grounded benchmarkwebpage diagnosticsVideo QAsemantic-logical failuresvisual-motion failuresstructural-temporal failuresfunctional failuresmulti-agent systemvideo-based diagnosis
Authors
Jiajun Xu, Yanghao Zhou, Jingyun Liao, Yu Bai, Jinxing Zhou, Chengliang Liu, Changsen Yuan, Bo Wang, Qian Liu
Abstract
Natural-language-driven "vibe coding" enables the one-shot generation of visually rich and interactive web applications, yet reliable assessment of their quality has not kept pace. Existing evaluations often score isolated artifacts or final task outcomes, offering limited evidence about which failures occur and why. We introduce VideoVIBE, a video-grounded benchmark that transforms human-operated webpage recordings into fine-grained diagnostic tasks. It contains approximately 1.7K diagnostic Video QA instances derived from 6,338 verified failures across generated webpages, spanning semantic-logical, visual-motion, structural-temporal, and functional failures. Diagnoses are grounded primarily in recorded presentation and behavior, with webpage source code used as complementary context. We further propose V2Lens, a training-free, evidence-grounded multi-agent system that challenges and selectively refines initial video-based diagnoses through targeted visual and source-code verification. Across thirteen closed-source and open-weight Video MLLMs, Gemini-2.5-Flash is the strongest standalone model with a score of 64.54, while V2Lens reaches 71.72, an improvement of 7.18 points. Together, our results show that video-grounded evaluation can move beyond isolated artifacts and aggregate outcomes toward a behaviorally faithful and diagnostically informative account of generated application quality.