Founref improves depth accuracy and speed using sparse metric anchors
FounRef: Robust, Structure-Preserving, and Fast Metric Refinement of Frozen Monocular Foundation Priors with Sparse Anchors
Computer Vision and Pattern Recognition
Summary
Getting accurate 3D depth from single camera images is hard because computers often struggle to know the real size of objects. The authors developed FounRef, a method that improves depth guesses by combining general depth knowledge from AI models with a few real-world distance points from sensors like LiDAR. This approach fixes errors without needing new training and works fast in many situations. It gives better and more reliable 3D measurements for things like robots or cameras looking at new scenes.
What this means in practice
- •For robotics engineers: Enable robots to obtain fast, accurate 3D depth maps from single cameras assisted by sparse sensor data for better navigation and interaction in new environments.
- •For augmented reality developers: Create more realistic AR experiences by refining depth maps combining AI priors and sparse metric data for different cameras and scenes.
- •For autonomous vehicle software teams: Improve perception by aligning monocular depth predictions with sparse LiDAR measurements to reduce depth errors and noise efficiently.$Commercial implications: Supports sale of enhanced perception modules that fuse AI and sensor data for safer autonomous driving.
Authors
Dan Halperin, Mirko Mählisch
Abstract
Dense metric depth from cameras is essential to real-world 3D applications, yet achieving accuracy, faithful surface geometry, and fast inference simultaneously remains challenging. Monocular foundation models provide rich, transferable geometric priors but lack reliable metric scale, while depth-completion networks recover metric depth at the cost of geometric fidelity, cross-domain robustness, or speed. We present FounRef, a training-free method that aligns a frozen monocular foundation prior with sparse metric anchors to produce dense metric depth. FounRef is modular by design: its depth prior, anchor source, and refinement solver can each be replaced independently. We instantiate FounRef with MoGe-2 and LiDAR anchors. FounRef validates each anchor against the prior's dense depth prediction, rejecting inconsistencies caused by cross-sensor misalignment that geometry-only filters cannot detect. It then applies global and local metric corrections through a structure-preserving solver, retaining the prior's fine-grained geometry. FounRef requires no task-specific training and operates out of the box across unfamiliar cameras and scenes. On out-of-domain data, it delivers up to 24% lower depth error, 92% lower surface-normal noise, and almost 15x faster inference than DMD3C, a state-of-the-art depth-completion network. By decoupling metric alignment from geometry prediction, FounRef provides an accurate, geometrically faithful, and efficient approach to dense metric depth that can directly benefit from future advances in foundation models and metric sensors.