Text to video models evaluated for physical realism with scene questions
PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA
Computer Vision and Pattern Recognition
Summary
Text-to-video models can make realistic-looking videos but sometimes ignore how things should physically behave, like objects falling or colliding realistically. The authors created PhyMetric, a new test that checks if these videos follow the expected physical rules by asking yes/no questions about each scene based on known physics. This approach helps identify exactly where models fail, unlike traditional scores that give only overall quality ratings. Their tests show that asking specific physical questions works much better than vague ones for judging realism in generated videos.
What this means in practice
- •For video generation engineers: Use PhyMetric to test and improve physical realism in text-to-video models by diagnosing specific failure modes.
- •For vision-language model developers: Evaluate and refine VLMs as judges for physical correctness in generated video content with targeted question answering.
Authors
Siwei Meng, Xuerou Cai, Rui Hu, Ping Liu
Abstract
Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene. Existing benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1{,}115 scenes spanning 50 physical categories across six domains, paired with 8{,}920 answer-balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a critical component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection. Our project page is available at: https://mengsiwei.github.io/PhyMetric/.