Self-supervised depth learning improves with scene structure awareness

Learning to Discriminate Scene Structures Makes Self-Supervised Depth Learning Scalable

Computer Vision and Pattern Recognition

Summary

Estimating depth from a single camera without supervision often struggles when trained on very different types of scenes. The authors found that models tend to learn dataset-specific patterns instead of general 3D structures, which hurts performance on new data. They propose a new method called MaSS that separates structural features from other details and trains the model to recognize scene layouts, helping it generalize better to diverse environments. Tests on multiple datasets show MaSS consistently outperforms existing methods and works well even on totally new scene types.

What this means in practice

Authors

Mengtan Zhang, Yuchuan Cui, Yu Ma, Zizhan Guo, Wei Ye, Rui Fan

Abstract

Self-supervised monocular depth estimation is appealing for its potential to scale with unlabeled data, yet existing methods often degrade when trained on diverse scenes. This study finds that, under self-supervision, models tend to encode dataset-specific structural biases rather than transferable scene geometry, leading to cross-domain optimization conflicts. To address this issue, this study proposes the MaSS framework, aiming to make self-supervised depth learning scalable to diverse data. MaSS introduces a dedicated structural representation pathway explicitly decoupled from contextual features, which is guided by a novel scene-structure discrimination objective to learn generalized, layout-aware structural representations for depth decoding. Trained and evaluated on four diverse datasets, both individually and jointly, MaSS substantially outperforms prior self-supervised methods and is the first to consistently benefit from mixed-domain training. Zero-shot evaluation on eight additional datasets further demonstrates its remarkable generalizability, particularly on those entirely out-of-distribution scenes. The source code is publicly available at https://mias.group/MaSS.