World models show inconsistent predictions across video and text outputs
One Model, Two Physical Stories: Auditing Misalignment in Multi-Modal World Modeling
Artificial Intelligence
Summary
World models try to predict what will happen next in an environment using different kinds of outputs like videos and text descriptions. This paper finds that sometimes the video and text predictions from the same model don’t match each other or real-world physics. The authors studied these mismatches and developed ways to measure them. Their results suggest current models struggle to be consistent across all types of output while staying physically accurate.
What this means in practice
- •For simulation developers: Check simulation tools that generate both video and textual predictions to improve cross-modal consistency and physical realism.
- •For ai product teams: Evaluate multi-modal AI models for alignment issues when deploying in systems requiring consistent visual and textual understanding.
Authors
Geigh Zollicoffer, Minh Vu, Rajiv Ranasinghe, Manish Bhattarai
Abstract
World models, systems that generate what happens next given current environmental conditions, are increasingly being implemented with multi-modal generation in mind. However, generating multiple modalities simultaneously, such as visual simulations alongside physical state predictions in the form of text, introduces the risk of cross-modal inconsistency. Tested separately, both outputs may look convincing while still disagreeing: a model can calculate that a ball should rebound in one modality, then generate no rebound in another modality, to say nothing of diverging from real-world dynamics entirely. In this work we focus on two failures explicitly: \emph{Internal misalignment}, the disagreement between the world model's generated video and the same world model's prediction in a different modalities, and \emph{external misalignment} the disagreement between the world model's generation and an analytic physical environment. We derive common contracts of event, magnitude, timing, and construct a physics grounded pipeline to make comparisons measurable in both external and internal settings. We then ask whether progressively supplying the model's own contract (the A ladder for the internal setting) or a corrected physical contract (the B ladder for the external setting) closes the respective gaps. Across four mechanisms and 20 settings, we find that while language answers all 22 text probes correctly with respect to the true environment, the neutral video is often in disagreement, suggesting that the current unified backbones may not be capable of correct reasoning, internal consistency, and external physical fidelity all at once.