Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created Sci-VBench, a large test set to check how well video generation models can make scientifically accurate and reasoning-based videos across many fields like science and engineering. They designed a scoring system that both regular people and AI judges can use to agree with expert opinions. When testing 16 advanced models, they found that while visual quality was similar, the models differed a lot in how well they understood and correctly showed scientific ideas and cause-effect relationships. This means making videos look good doesn’t guarantee the scientific content is accurate.
video generationscientific reasoningbenchmarkevaluation protocolmachine learningcausal correctnessprompt groundingnatural sciencemultimodal large language modelsvisual realism
Authors
Diandian Zhang, Tingyu Song, Lin Fu, Zheyuan Yang, Yilun Zhao
Abstract
We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.