NeuIDO learns a unified model to predict object dynamics from video
NeuIDO: Neural Intrinsic Dynamics Operator for Physics-Informed 4D World Models
Computer Vision and Pattern Recognition
Summary
Predicting how objects move and interact in the world is important for machines to understand their surroundings. Current methods rely on hard-coded physical rules and don’t always capture real-world complexity. The authors propose NeuIDO, which learns the underlying rules of motion directly from videos without needing those fixed assumptions. This allows it to quickly predict object dynamics in new scenes it hasn’t seen before, and adapt with just a little extra data. Their experiments show NeuIDO can combine different visual examples into a shared understanding of how things move in 3D space over time.
What this means in practice
- •For robotics teams: Enable robots to infer and predict complex object movements in new environments directly from video input without explicit physics programming.
- •For augmented reality developers: Enhance AR systems by allowing real-time prediction of object interactions and dynamics from live video to improve interaction realism.
Authors
Jiajing Lin, Xin Zhang, Jianhua Sun
Abstract
World models aim to capture environmental dynamics and predict future trajectories, showing growing potential for embodied intelligence. Physics-informed 4D generation integrates physical simulation to predict 3D object interactions, offering a promising pathway toward world models. However, this paradigm relies on manually imposed dynamical assumptions rather than internalizing world dynamics, and thus still leaves a gap toward a true world model. To bridge this gap, we propose NeuIDO, a novel world dynamics modeling framework that learns a unified intrinsic dynamics representation from visual observations, advancing physics-informed 4D generation toward a world model. Specifically, we formulate world modeling as a neural operator learning problem and introduce a two-stage training strategy to learn a generalizable mapping from the visual observation distribution to the intrinsic dynamics distribution. Building on this observation-dynamics mapping, NeuIDO enables zero-shot dynamics inference directly from videos and can be further aligned with complex real-world dynamics via few-shot adaptation. Extensive experiments demonstrate that NeuIDO effectively unifies the intrinsic dynamics underlying diverse visual observations into a shared representation and rapidly infers dynamics in novel scenes.