PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose PixWorld, a new model that combines 3D reconstruction and generation into one system by working directly with images instead of hidden features. Unlike past methods that used complicated encoded spaces, PixWorld learns from actual rendered images, which helps it keep more 3D details. They also introduce a special loss that helps the model understand 3D shapes better by comparing features from a 3D-aware model. PixWorld performs better than earlier methods that used latent spaces and is as good as top methods for reconstructing 3D scenes.
3D reconstruction3D generationlatent diffusionpixel-space diffusionrendered imagesVariational Autoencoder (VAE)Representation Autoencoder (RAE)geometry perception loss3D foundation model3D scene fidelity
Authors
Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Jia-Wang Bian
Abstract
3D reconstruction and generation are commonly tackled by separate paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent works attempt to unify them in latent space, but with notable drawbacks: the diffusion objective is defined on latent features rather than the underlying 3D representation, and both branches suffer from information loss introduced by latent encoding, while requiring a pretrained Variational Autoencoder (VAE) or Representation Autoencoder (RAE). In this paper, we reformulate these two tasks under a unified pixel-space diffusion paradigm and introduce PixWorld, a single model that jointly addresses 3D reconstruction and generation. By supervising diffusion directly on rendered images, PixWorld removes the above limitations and aligns optimization with 3D scene fidelity. Beyond photometric and perceptual supervision that operates at the 2D image level and lacks 3D geometric awareness, we further introduce a geometry perception loss that aligns rendered views with their ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision. PixWorld consistently outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods, demonstrating the superiority of a unified pixel-space approach.