Vision language models fall short on understanding physical events and dynamics
PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop
Computer Vision and Pattern Recognition
Summary
Some computer models can understand pictures and words together, but it’s unclear if they really get how things move and interact in the real world. The authors created PhysVista, a test that checks if these models see, think about, and judge physical events correctly, just like humans do. Their tests show many models struggle to connect visual details with real-world physics, especially when watching videos showing things happening. This highlights a need for better ways to help AI understand the physical world more deeply.
What this means in practice
- •For multimedia application developers: Test and improve visual reasoning modules in applications using videos with physical interactions to enhance realism and user experience.
- •For game developers: Evaluate AI components that judge physical plausibility in generated animated scenes to create more believable virtual environments.
Authors
Xinge Peng, Yiting Lu, Tianwu Zhi, Wen Wen, Jianzhao Liu, Xin Li, Zhibo Chen
Abstract
Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities, yet whether they truly capture the physical consistency underlying real-world dynamics remains unclear. Existing benchmark paradigms often suffer from fragmented evaluation, focusing on isolated cognitive stages while overlooking the inherent synergy between perception, reasoning, and physical judgment. The lack of a holistic perspective limits the ability to diagnose whether VLMs can reliably evaluate the physical authenticity of emerging generative models. To address these issues, we introduce PhysVista, a benchmark designed to evaluate physical intelligence in VLMs through a closed cognitive loop framework inspired by the human seeing-reasoning-assessment process. PhysVista restores this loop by jointly evaluating physical state perception, physical dynamics reasoning, and physical plausibility assessment. It further distinguishes event-level reasoning and scale-level reasoning to enable fine-grained analysis of physical understanding. In addition, PhysVista incorporates both real-world and AI-generated videos, allowing evaluation across diverse domains and emerging generative scenarios. Extensive experiments across a diverse set of VLMs reveal substantial limitations in physical reasoning and plausibility assessment, highlighting a persistent gap between visual recognition and genuine physical understanding, and pointing toward more principled designs for physically grounded multimodal intelligence.