Summary
Understanding and labeling 3D scenes usually needs lots of trained models and data, especially to recognize many different items. The authors developed GoDeep, a method that uses a vision-language model to turn images into detailed text descriptions instead of relying on 3D data or training. These text descriptions are combined and analyzed in a language-based space to identify objects, even ones that the system has never seen before. This approach not only competes well with other methods but also makes it easier to explain what the system is detecting in the scene. Additionally, GoDeep can precisely locate these objects in 3D without needing specific 3D annotations.
3D semantic segmentationopen vocabularyCLIP featuresvision-language modellanguage embedding spaceannotation-free learningScanNet++out-of-vocabulary objectsscene understandingexplainability
Abstract
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.