OmniPoint builds 3D maps from any single camera image
OmniPoint: Universal Monocular Metric Pointcloud from Any Camera
Computer Vision and Pattern Recognition
Summary
Getting accurate 3D pictures from just one camera can be tricky because cameras capture scenes in different ways. The authors created OmniPoint, a method that works with many camera types, like normal, fisheye, or 360-degree cameras. Instead of guessing depth in a fixed way, OmniPoint separates how the camera sees the scene from the actual 3D info, making it more flexible. It also uses clever tricks to learn from limited data and use extra inputs without confusing the system. Tests show it works well without extra training on new cameras.
What this means in practice
- •For mobile app developers: Create apps that generate accurate 3D maps from images taken by any smartphone camera model.$Commercial implications: Enables selling 3D scanning apps compatible with varied phone cameras using one flexible method.
- •For robotics engineers: Build robots that perceive metric 3D environments from a range of camera types without retraining for each.
Authors
Botao Ye, Marc Pollefeys, Ming-Hsuan Yang, Abhijit Kundu
Abstract
Recovering metric 3D geometry from monocular images is a fundamental computer vision task, yet current methods remain heavily fragmented by fixed camera model assumptions and inflexible input schemes. We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors, including pinhole, fisheye, and equirectangular projections, while accommodating varying geometric priors. To overcome projection rigidity, OmniPoint abandons conventional planar depth regression. It instead adopts a decoupled ray and distance representation alongside a decoupled training objective, explicitly separating the camera projection model from the scene structure. To address the severe scarcity of training data for alternative cameras, we introduce a bidirectional augmentation strategy that explicitly bridges labeled perspective data and unlabeled omnidirectional domains in 3D space. Furthermore, to seamlessly integrate optional inputs like camera intrinsics or sparse depth without destabilizing the network through feature distribution shifts, we propose a robust information injection mechanism. This mechanism utilizes learnable input state embeddings to resolve architectural ambiguity and applies vectorized Gaussian smoothing to densify irregular measurements. Extensive experiments demonstrate that OmniPoint achieves state-of-the-art zero-shot performance across multiple benchmarks, establishing a robust new standard for unified monocular 3D reconstruction.