Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
2026-09-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created Puffin-World, a system that combines understanding of physics, 3D shapes, and appearances to build and explore virtual worlds. It uses a special camera approach to handle different views and motions, making the generated worlds look consistent and realistic over time. Their method predicts how physical changes happen in the future and links images directly with 3D geometry together. They also made a large dataset to help the system learn and shared all their work to support more research. This allows the system to explore and understand virtual environments without needing extra tools.
multimodal architecture3D reconstructionphysical simulationgeometryappearance modelingOmni-cameraworld generationvision-language datasettrajectory dataclosed-loop exploration
Authors
Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy
Abstract
We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.