Physical video benchmark measures realism of generated videos
PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
Computer Vision and Pattern Recognition
Summary
Many AI models create videos that look good but might not follow real-world physical rules like movement or object interactions. The authors created PVBench, a benchmark to measure how well generated videos obey physical laws, using 46 detailed tests. They check things like object shapes, movements, and whether conservation laws hold in the videos. Their method uses special vision models to make precise physical measurements, not just human guessing. They also tested popular video-generation models to see how they perform on these physical tests.
What this means in practice
- •For video game developers: Automatically verify that generated in-game animations obey physical laws across multiple object types and interactions.
- •For robotics simulation engineers: Evaluate generated robot motion videos for accurate physical and kinematic realism using object-level measurements.
Authors
Kaiwei Chen, Moran Li, Zhenye Gan, Yabiao Wang, Chengjie Wang, Kexin Zhang, Yong Liu, Jiangning Zhang
Abstract
Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.