World models learn scene physics from short exploratory clips quickly
PhysiCalWorld: In-Context Physics Calibration for World Models
RoboticsArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Predicting how objects move and interact in new scenes is hard because their physical properties like weight and friction aren’t directly visible. The authors introduce PhysiCalWorld, which watches a short video of how things move when pushed around and uses that to understand the scene’s unique physics. This helps the model make better predictions about what will happen next without needing to be retrained or know complicated physics formulas. Their method improved accuracy and planning across various test environments just by learning from the visual data.
What this means in practice
- •For robot engineers: Improve robots’ ability to predict object behavior in new environments by quickly adapting to their unique physical properties from short interaction data.
- •For game developers: Increase accuracy of physics-based predictions in games for diverse environments without manual tuning of physical parameters.
Authors
Yunhao Luo, Wentao He, Yinpei Dai, Hyelin Nam, Hongze Fu, Joyce Chai, Nima Fazeli, Jeong Joon Park
Abstract
Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.