Latent 3d gaussian memory enables fast spatially consistent infinite worlds
L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory
Computer Vision and Pattern Recognition
Summary
Generating virtual worlds that stay consistent as you explore them is hard, especially if you want the world to update in real time. The authors introduce a new way to remember and represent 3D scenes using special blobs called latent 3D Gaussians that grow with the world size, not how long you explore. Their method lets the system create detailed, consistent imagery quickly without needing to reprocess images repeatedly, making it fast enough for real-time use. Experiments show their approach is much faster and uses fewer resources than earlier methods.
What this means in practice
- •For game developers: Build infinite virtual worlds that update visually in real time with stable spatial details during long player exploration.$Commercial implications: Enables faster and more resource-efficient real-time game worlds that retain consistency, enhancing user experience with scalable content.
- •For virtual reality creators: Create immersive VR environments where spatially consistent scenes update quickly as users move through large spaces.
Authors
Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim
Abstract
Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to $38.6\times$ fewer 3D primitives, $3.2\times$ faster memory updates, and $359\times$ faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.