Marigold V2 improves depth maps from single camera images with smarter AI models
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Figuring out how far away things are from a single picture is really hard for computers, and current methods often miss fine details. The researchers behind Marigold V2 looked at how AI models originally made for generating images can be adapted to improve this task. They found ways to make the model understand important features better and fine-tuned it with a special technique, leading to sharper and more accurate depth results. Their system also works well even on unfamiliar pictures and can help with other detailed image tasks like figuring out surface angles and separating shadows from colors.
Monocular depth estimationDiffusion transformerComputer visionFlow-matching modelsQuantizationFine-tuningSinkhorn lossOut-of-distribution generalizationSurface normal estimationIntrinsic image decomposition
Authors
Igor Pavlovic, Thiemo Wandel, Anton Obukhov, Luca Bartolomei, Andrey Davydov, Fabio Tosi, Matteo Poggi, Sabine Süsstrunk, Dengxin Dai
Abstract
Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web