3D geometry learning improves by shifting asymmetry to representation space
Rethinking Asymmetry in Self-Supervised 3D Geometry Learning
Computer Vision and Pattern Recognition
Summary
3D geometry learning from images needs a trick called 'asymmetry' to avoid easy but incorrect answers. The authors found that placing this asymmetry not in the input images but inside the 3D scene representation helps the computer understand depth and camera positions better. They also use different groupings of views and align their features, and even include a frozen vision model to guide learning, all without needing labeled 3D data. This new way of training models makes them better at figuring out 3D shapes from photos and helps improve later supervised training.
What this means in practice
- •For computer vision engineers: Build systems that estimate 3D scene structure from unposed images with better depth and camera pose accuracy without labeled 3D data.
- •For augmented reality developers: Improve 3D environment understanding from images to support occlusion and interaction in AR applications without manual 3D annotations.
Authors
Gyeongjin Kang, Freeman Cheng, Botao Ye, Weijie Lyu, Ming-Hsuan Yang, Eunbyung Park
Abstract
Self-supervised learning relies on asymmetry between what a model observes and what it must explain. Recent methods learn 3D geometry from unposed images by predicting cameras and scene structure, then rendering held-out views. To avoid trivial solutions, they impose this asymmetry in the input space, restricting how views interact through separate branches or masked attention. We revisit this design with geometric accuracy, rather than rendering quality, as the main objective. When a held-out view lies between context views, the context already constrains the geometry it observes. We therefore impose the asymmetry in the 3D representation space, excluding held-out views only from the reconstructed scene while a shared backbone uses all views for correspondence. This improves depth and pose estimation over input-space designs. We further exploit view density: sparse and dense observations depict the same scene but provide different amounts of correspondence. Aligning sparse-context features with dense-context features supplies a geometric training signal alongside rendering, without a pretrained geometric teacher. We additionally align features with a frozen vision foundation model, which brings further gains. None of these signals requires geometric annotation, and the resulting model improves geometry estimation over prior self-supervised methods. Controlled experiments isolate the contribution of each component, and self-supervised pretraining also improves subsequent supervised training.