Model reconstructs 3D scenes from any type of central camera
Any3R: Feedforward 3D Reconstruction from Any Camera Type
Computer Vision and Pattern Recognition
Summary
3D reconstruction from pictures usually requires cameras with specific lenses, like pinhole cameras, to work well. The authors developed a method called Any3R that can take images from any central camera type, no matter the lens or distortion, and still figure out camera positions and 3D shapes accurately. They do this by focusing on the direction of rays through each pixel, rather than the lens details, enabling the model to handle mixed camera types without retraining. Their method performs well on different datasets with various cameras, making it useful for building 3D models from diverse photo collections.
What this means in practice
- •For photogrammetry teams: Reconstruct 3D models from crowdsourced images taken with varied camera types without special camera calibration or retraining.
- •For archival digitization teams: Create scalable 3D reconstructions from historical image archives shot with diverse cameras, improving preservation and analysis.
Authors
Jingong Chen, Qingwen Zhang, Youming Deng, Chulwoo Pack, Olaf Wysocki, Boris Jutzi, Kwanghee Won, Kyle Gao
Abstract
Geometry foundation models recover camera poses and dense geometry from a set of images with a feed-forward network, but they expect pinhole images. That assumption is inside the backbone too: the positional encoding and the training loss both bind an image position to a direction in space, and both depend on the lens. We condition both on the dense ray field, the unit direction through every pixel, which describes any central camera without a lens type, distortion coefficients or a bound on field of view. Angular coordinates give the rotary encoding the measured ray angle of each patch, evaluated per view, and solid-angle weighting scales the point loss by the area each pixel covers on the sphere, so neither the encoding nor the supervision depends on the lens. Any3R is therefore a feed-forward model that takes any central camera, and is robust to mixing many camera types per scene, with no retraining per lens. On a benchmark of thirteen central cameras across three datasets, where every method reconstructs with its own predicted camera poses, it is the most consistently accurate method. Any3R enables scalable 3D reconstruction from heterogeneous crowdsourced and archival imagery collections sourced across diverse camera types. Project page: https://jingong-chen.github.io/Any3R/