Model improves 3D depth estimates from multiple car cameras
CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding how far away things are is important for self-driving cars. The problem is that cameras placed around a car often don't have much overlap in their views, so the model has to guess depth from single images. The authors created a method that helps the computer take into account differences in camera types and uses information from nearby cameras in smart ways based on geometry. This makes depth estimates more accurate and consistent, and it works without needing labeled training data. Their approach was tested on driving datasets and did better than previous methods.
3D depth estimationmulti-view camerasself-driving carsmonocular cuescamera intrinsicscross-image attentionphotometric consistencyself-supervised learninggeometry constraints
Authors
Samer Abualhanud, Max Mehltretter
Abstract
Reliable 3D understanding of the surrounding environment is a core requirement for autonomous driving. Multi-view surround camera rigs provide broad scene coverage, but the spatially adjacent images typically overlap only minimally. Consequently, the depth of most pixels must be inferred from monocular appearance cues. These cues can appear differently across images and may therefore be interpreted differently by the depth estimation model. We target two main sources of cross-image inconsistency: differences in camera intrinsics and the limited receptive field of each image. We address the former by conditioning the features on per-pixel camera-aware ray embeddings, enabling the network to account for camera-dependent variations in monocular cues. We address the latter by extending each pixel's context beyond its own image through cross-image attention constrained to geometrically plausible regions, derived from the calibrated rig setup. The model is trained in a fully self-supervised manner based on photometric consistency. Evaluations on DDAD and nuScenes show improved overall depth accuracy and cross-image depth consistency over state-of-the-art self-supervised methods under in-domain and cross-domain evaluation. Code is available at https://abualhanud.github.io/CrossDepthPage/.