Monocular depth models face challenges in viewpoint consistency and metric anchoring
Depth Anyviewpoint? Evaluating Monocular Depth Over Viewpoint Sequences
Computer Vision and Pattern Recognition
Summary
Predicting how far away things are from a single photo is useful but tricky because the output often only shows relative distances, not exact ones. The authors studied how well these models keep consistent distance predictions when the camera moves around a scene in a video. They created a new way to test this and found that current models struggle to keep depth predictions steady over time and changing views. They also found that using techniques that reconstruct 3D scenes from video to set exact distance scales is not simple. This work highlights the challenges that remain for monocular depth models to be reliable in real-world, dynamic settings.
What this means in practice
- •For video processing engineers: Improve validation techniques for depth estimation from single images over scenes captured by moving cameras to better handle video input consistency.
- •For augmented reality developers: Develop augmented reality apps that require steady depth maps by addressing viewpoint and temporal inconsistencies identified in monocular depth models.
Authors
Theo Rode, Rui Zhang, Ryan Aubrey, Olivia Li, Jessica Huang, Liv Chu, Healani Dowd, Calden Wloka
Abstract
Monocular depth estimation aims to predict depth maps from single RGB images (forgoing the need for stereo camera pairs) and has seen impressive leaps in performance in recent work due to the introduction of affine-invariant losses and the use of foundation model features. These monocular depth models predict affine-invariant depth, which describes the relative positions of objects in the scene instead of absolute depth. While these models demonstrate strong performance on single image benchmarks and qualitatively consistent predictions in video, their effectiveness at preserving consistent spatial structure has been less thoroughly explored. Many potential real-world applications, such as handling video streams as input, rely on coherent three dimensional consistency in scenes. Further, anchoring relative depth predictions to a metric reference for a given scene is a key challenge for applying monocular depth models to real-world situations. We develop a novel evaluation protocol for monocular depth models using video sequences with moving camera positions to test the consistency of prediction across variation in viewpoint. We demonstrate this methodology over diverse, real-world video sequences where we are able to anchor depth model output to metric depth values using structure from motion. Our work demonstrates open challenges in the development of broadly applicable monocular depth foundation models, particularly with their ability to provide temporal consistency over long time frames. We also identify challenges in the straightforward adoption of structure-from-motion as a metric grounding tool for monocular depth estimation models.