Benchmark measures how videos show physical world details
World Embedding Benchmark
Computer Vision and Pattern RecognitionComputation and Language
Summary
Understanding how videos capture real-world physical properties is tricky. The authors created a large set of videos paired with detailed physical data from simulated scenes. They used this set to test how well current video understanding models recognize and recover physical information. They found that while some models can match videos with descriptions, they struggle to grasp precise physical details. Improving model training helps with some tasks but makes others harder, showing a trade-off in video understanding.
What this means in practice
- •For video generation developers: Use physical video embeddings to retrieve reference videos that enhance the physical realism of generated videos.
- •For machine learning engineers: Improve video understanding models by evaluating and balancing physical alignment with quantifiable physical property recovery.
Authors
Yiqi Liu, Ruifeng Yuan, Yang Wang, Long Li, Fengyu Cai, Hou Pong Chan, Jialin Yu, Hao Zhang, Chenghua Lin, Chenghao Xiao
Abstract
Physical fidelity has received increasing attention in world models and video generation, yet how video representations encode physical information remains less understood. We introduce the World Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification. We use these tasks to distinguish cross-modal physical alignment from the recoverability of quantitative physical information. Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings. Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability. Finally, we use the embeddings to retrieve reference videos for retrieval-augmented generation with MiniMax-H3. Retrieved references improve the physical fidelity of generated videos, with stronger retrieval models yielding larger gains in our experiments. Together, these findings highlight the need to evaluate physical alignment and property recoverability jointly, and demonstrate the utility of physical representations for improving video generation.