OmniRay predicts 3D scenes from any camera image directly
OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields
Computer Vision and Pattern Recognition
Summary
Most methods that turn a single photo into a 3D shape assume the camera acts like a simple pinhole, which distorts wide-angle pictures. The authors developed OmniRay, a model that learns to understand how rays travel from many types of cameras, including fish-eye and panoramic lenses. It breaks down the camera's ray directions into simple parts, allowing it to handle complex and unusual camera shapes without needing camera info during use. This leads to better 3D reconstructions from just one image, no matter the camera type.
What this means in practice
- •For augmented reality developers: Create accurate 3D models from images taken by diverse mobile device cameras without needing camera specifications.
- •For robotics engineers: Improve environment perception by reconstructing 3D scenes from single wide-angle camera images regardless of lens type.
Authors
Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa, Anand Bhattad
Abstract
Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models remove this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry can be viewed as having a hierarchy of complexity for representing rays. We present OmniRay, a feed-forward model whose Ray Module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form residual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse camera geometries. Given a single image, OmniRay predicts a ray direction and an up-to-scale radial distance for every pixel, from which a 3D point cloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.