Lang3DSeg uses point transformers for open 3D LiDAR segmentation
Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers
Computer Vision and Pattern RecognitionRobotics
Summary
Understanding and labeling 3D scenes accurately is crucial for safe autonomous driving, but manually labeling each point in 3D scans is costly. The authors present Lang3DSeg, a system that transfers knowledge from 2D vision-language models to 3D LiDAR data without needing manual annotations. They overcome errors caused by projecting 2D labels onto 3D points by applying a depth-aware correction method. Their approach works outdoors in real-time and surpasses previous annotation-free methods on standard benchmarks.
What this means in practice
- •For autonomous vehicle developers: Enable real-time 3D perception systems that label objects in outdoor LiDAR scans without costly manual annotations.$Commercial implications: This enables more scalable and adaptable autonomous driving perception systems that require less manual data labeling.
- •For robotics system engineers: Build outdoor robots with improved scene understanding using annotation-free 3D semantic segmentation from point cloud data.
Authors
Cigdem Kokenoz, Amir Salarpour, Alkim Domeke, Christopher Salas, Pedram MohajerAnsari, Long Cheng, Mert D. Pesé, Bing Li
Abstract
Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.