Video models struggle with thinking tasks despite good visuals
From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Making videos with AI has gotten visually impressive, but these models often can't follow rules or goals in the videos they create. The authors designed a new way to test whether video AI truly reasons with the content instead of just looking good. They found that many models fail on logic and rule-based challenges. To help, they made a tool that improves how instructions are given to these video models, leading to better reasoning without changing the model itself.
What this means in practice
- •For video generation teams: Improve video AI outputs by refining prompts to enhance logical and goal-based reasoning in generated content without altering underlying video models.
- •For multimodal ai developers: Use the benchmark to evaluate and diagnose reasoning capabilities of video-generative systems along multiple cognitive dimensions.
Authors
Meng Luo, Yicheng Liu, Jiahao Wang, Yuanxing Zhang, Xin Tao, Pengfei Wan, Kun Gai, Hao Fei
Abstract
Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--executing symbolic rules, respecting physical laws, and pursuing intentional goals--remains an open question. Existing benchmarks only partially address this, often conflating visual quality with cognitive correctness. We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks. To enable precise diagnosis, we design a three-level VLM-as-Judge protocol that independently assesses video-level fluency, task-level rule adherence, and sample-level goal realization. Evaluations of leading models reveal a striking gap: while models achieve strong rendering scores, they consistently fail on logic-heavy and rule-constrained tasks. To address this, we propose Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM. Trained via reinforcement learning with purely text-based rewards, Vid-PRE produces concise, constraint-aware prompts without the instability of video-level reward signals. Experiments show that Vid-PRE yields substantial reasoning improvements across multiple generators without architectural modifications. Together, VWG-Bench and Vid-PRE offer a rigorous diagnostic lens and a scalable path toward true think-with-video capabilities. All data and code are publicly available at https://huggingface.co/datasets/KlingTeam/VWG-Bench.