GoDeep maps 3D scenes using language without 3D training data
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding 3D scenes usually requires training on lots of labeled 3D data or special 3D tools. This paper shows a way to describe and identify parts of a 3D scene using just language descriptions from 2D images, without needing 3D training or special 3D models. The authors use a language model to translate image content into text descriptions that are then matched directly into a language space, helping to detect and locate objects even if they were never seen before. This method also lets users understand why a certain object was labeled at a particular spot, making the approach explainable.
What this means in practice
- •For 3d mapping teams: Generate detailed 3D scene labels without any 3D training data by leveraging language-based descriptions from posed images.
- •For cultural heritage conservators: Locate and identify rare or unseen objects in 3D scans of heritage sites using language-driven scene understanding.
Authors
Thodoris Betsas, Anastasios Doulamis, Andreas Georgopoulos
Abstract
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.