AEON combines video and 3D models for realistic 4D world simulation
AEON: Unifying Video and 3D World Models
Computer Vision and Pattern Recognition
Summary
Making computers understand and recreate the world around us can happen in two ways: by generating realistic videos or by building stable 3D models. The video approach looks real but can mess up how things fit together in 3D space. The 3D approach keeps shapes right but looks less real and often static. The authors created AEON, which mixes the two to make worlds that look real and keep their 3D shapes over time, like a 4D model that moves and changes. This helps simulate environments better for things like games or virtual reality.
What this means in practice
- •For game developers: Create dynamic game worlds that look realistic and maintain 3D consistency over time using combined video and 3D modeling techniques.$Commercial implications: Enables selling immersive interactive game environments with both realism and geometric accuracy.
- •For virtual reality engineers: Build persistent and visually accurate 4D environments for VR simulations that support natural scene changes and realistic views.
Authors
Weiqi Zhang, Wenyuan Zhang, Yu-Shen Liu, Junsheng Zhou
Abstract
Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON