StrucPhysVideo models physical object motion from captions and robot actions
StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions
Computer Vision and Pattern Recognition
Summary
Understanding how objects move and interact like bumping or changing shape is hard for computers. The authors created StrucPhysVideo, a method that learns from videos with detailed captions about objects and their physical events. It separates camera movement from object motion so the computer focuses on real dynamics. Their models can predict what will happen next in a video either from text descriptions or from robot commands. This helps computers better imagine and plan physical interactions in the real world.
What this means in practice
- •For robotics engineers: Generate visual predictions of object motions resulting from robot arm commands to improve planning and control.
- •For computer vision developers: Create video models that better understand physical interactions by training on data with explicit object and event annotations.
Authors
Awomo-WM Team, :, Enhui Ma, Kaiwen Guo, Tingrui Zhang, Wei Song, Yingshui Tan, Jianhua Xu, Tong Zhang, Kaicheng Yu
Abstract
Modeling physical dynamics, including how objects move, interact, and change state, is central to video world models for embodied AI. We present StrucPhysVideo, a family of video world models that bridges physics-focused data curation with language- and action-conditioned prediction of scene evolution. Our data pipeline combines motion-aware video segmentation, quality and content filtering, and physical relevance verification with structured annotations of objects, materials, and temporally localized interactions. By disentangling camera motion from object behavior and explicitly describing contact, deformation, and state transitions, the pipeline provides supervision grounded in observable physical events. Building on these data, we introduce StrucPhysVideo-TI2V, a sparse Mixture-of-Experts (MoE) text-image-to-video model trained with a curriculum that progressively emphasizes physical dynamics while retaining general-domain video data. StrucPhysVideo-TI2V achieves state-of-the-art performance on Physics-IQ Verified, scoring 45.5% and outperforming Cosmos3-Super-Image2Video by 2.8 percentage points. Caption ablations across backbones further demonstrate the effectiveness of physics-focused supervision. We further extend StrucPhysVideo-TI2V to StrucPhysVideo-IA2V, an interactive image-action-to-video world model that predicts visual outcomes from robot end-effector commands. Action conditioning, causal autoregressive generation, and few-step distillation enable incremental robot rollouts with only four denoising steps. Together, StrucPhysVideo advances physical dynamics modeling from image- and language-conditioned video prediction toward action-driven interaction.