Depth Pro 2 improves sharpness and consistency in depth maps

Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

Computer Vision and Pattern RecognitionMachine Learning

Summary

Estimating how far away things are from a single camera image is hard, especially when you want the results to be clear and stable over time in videos. The authors found that existing methods usually trade off between making the depth image sharp or making it consistent between frames. They created Depth Pro 2, a single neural network that works for both still images and videos, producing depth maps that are accurate, sharp at edges, and smoothly consistent over time. Their method outperforms previous work in all these aspects.

What this means in practice

  • For augmented reality developers: Create AR experiences with accurate and stable 3D depth perception from single cameras, improving user immersion and interaction.$Commercial implications: Enables new or enhanced AR apps and devices that require precise real-time depth estimation for single-camera setups.
  • For robotics engineers: Integrate reliable monocular depth maps into robot navigation systems to enhance obstacle detection and environment mapping.

Authors

Peiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun

Abstract

We present Depth Pro 2, an approach to estimating dense metric depth for images and video at full resolution. By analyzing prior methods for monocular depth estimation, we find that they commonly trade off temporal consistency and boundary sharpness: multi-frame methods favor consistency, single-frame methods favor sharpness. To overcome this trade-off, we train a single, scalable network that processes both images and video and predicts depth maps at native resolution. Depth Pro 2 produces depth maps that are metrically accurate, sharp, and temporally consistent all at once, outperforming prior work in all of these dimensions. We release a family of models that define a new Pareto frontier in terms of metric accuracy, sharpness, temporal consistency, and runtime. We release code and weights at https://apple-aiml-research.github.io/ml-depth-pro-2/