Phyprobe improves checking physical realism in generated videos
PhyProbe: Rethinking Physical Consistency Evaluation in Generated Videos
Computer Vision and Pattern Recognition
Summary
Checking if videos made by computers behave realistically is hard. Current methods either use generic tools that miss physical details or get stuck on specific examples. The authors created PhyProbe, a tool that learns to score how physically consistent a video is by combining different types of guidance. PhyProbe works better than previous methods, especially on hard cases where videos don’t match well or are both computer-generated. It also aligns well with what people think looks physically correct, even when tested on preferences beyond physics.
What this means in practice
- •For video content developers: Automatically assess and improve physical realism in AI-generated videos to enhance visual quality and user trust.
- •For automated video quality teams: Integrate a reliable physical consistency score into video quality pipelines to detect artifacts missed by conventional evaluators.
Authors
Max Ku, Jiaojiao Fan, Zekun Hao, Francesco Ferroni, Heng Wang, Wenhu Chen, Ming-Yu Liu, Prithvijit Chattopadhyay
Abstract
Evaluating the physical consistency of generated videos remains a fundamental challenge. Existing approaches rely on off-the-shelf vision-language models, which can often be myopic to physical dynamics, or fine-tuned evaluators trained on human annotations, which overfit to dataset-specific cues and fail to generalize. A key challenge is that existing supervision sources provide either relative ordering or absolute scores, but not both reliably and consistently across varied settings. To this end, we introduce PhyProbe, an evaluator that extracts features from a frozen pretrained spatio-temporal encoder and maps them to a scalar physical consistency violation score via a lightweight scoring head. PhyProbe is trained through a unified objective combining pairwise ranking, regression on noisy scalar annotations, and anchor-based calibration over a curated set of heterogeneous supervision sources. Experiments show that PhyProbe outperforms prior methods on most pairwise benchmarks spanning real-generated and generated-generated pairs under varying correspondence, with the largest gains in no-correspondence and generated-generated settings where existing fine-tuned evaluators degrade sharply. PhyProbe achieves strong correlation with human judgments, with close agreement between rank-based and linear metrics, indicating that scores are both well ordered and anchored to a stable [0, 1] scale. Further, despite being trained on supervision indicative of physical consistency, without explicit general-preference labels, PhyProbe also performs competitively on human preference benchmarks: consistent with the observation that physics violations are entangled with broader quality degradations.