Geometry native latent space improves 3d consistent scene generation
GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation
Computer Vision and Pattern Recognition
Summary
Creating 3D scenes that look right from all angles is hard because most computer programs focus on appearance instead of true 3D shape. The authors offer a new way to represent scenes that naturally captures their 3D structure in a compact form. This lets computers generate realistic images that also stay consistent across different viewpoints. Their method improves both the quality of generated images and the accuracy of the 3D scene structure compared to older techniques.
What this means in practice
- •For augmented reality developers: Generate 3D-consistent scenes that improve realism and interaction fidelity in AR applications.
- •For 3d animation studios: Create photorealistic animations with better 3D coherence between frames for smoother visual effects.
Authors
Jiahao Lu, Minghao Yin, Wenbo Hu, Hengyu Liu, Wang Zhao, Sai-Kit Yeung, Ying Shan, Yuan Liu
Abstract
We present a compact geometry-native latent space as a shared foundation for perception and generation. Visual generators can produce photorealistic frames without preserving a consistent 3D scene. We argue that this is not only a modeling problem but also a representation problem: generators typically evolve appearance-centric latents, while perception models recover geometry in a semantically rich space that encodes cross-view structure. Rather than adding geometry as another output, we reparameterize a geometry foundation model's features into a compact latent space for generation. We realize this shift with the geometry-native autoencoder (GAE), whose latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks. In controlled comparisons that hold the generator and training protocol fixed, replacing the latent with GAE improves both visual quality and independently measured 3D coherence: FVD falls by $12.7\%$ and $23.1\%$ on RealEstate10K and DL3DV, and camera-trajectory error is halved on RealEstate10K. Together, these results show that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.