Video world model improves long horizon scene consistency
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphics
Summary
Understanding how environments change in videos over time and from different angles is a challenge. The authors developed WorldCrafter, which stores memories of a scene in three dimensions without needing explicit depth information. This helps the model keep track of what it saw before, even after long periods and viewpoint changes. It can generate consistent video scenes from a single image or text, making exploration more accurate and visually coherent.
What this means in practice
- •For game developers: Create interactive game environments that maintain consistent visuals and camera control during long play sessions.$Commercial implications: Enables more immersive and coherent dynamic scenes in games, improving player experience and control, a valuable commercial feature.
- •For robotics engineers: Build robots capable of better spatial memory and scene understanding for navigation and manipulation across dynamic environments.
Authors
Wangbo Yu, Kunhao Liu, Wenbo Hu, Shenghai Yuan, Chaoran Feng, Haiyang Zhou, Yukun Huang, Yiran Wang, Wang Zhao, Yingmin Luo, Ying Shan
Abstract
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.