LiDAR diffusion models learn 3D features from 2D images
Geometry Without Coordinates: LiDAR Diffusion as a 3D Feature Bridge
Computer Vision and Pattern Recognition
Summary
Working with 3D LiDAR data is hard because there isn’t much labeled data to train big AI models directly on it. The authors created a new model that uses ideas from 2D image AI models by training on labels generated from 2D images instead of real 3D labels. Their model can predict different 3D scene info like depth and object segments from LiDAR points by using text prompts. They found the model learns meaningful 3D features even without using raw coordinates, suggesting it transfers knowledge from 2D images to 3D shapes effectively.
What this means in practice
- •For autonomous vehicle engineers: Improve 3D scene understanding from sparse LiDAR by using models trained with 2D image labels, enhancing perception without extensive 3D annotations.
- •For robotics system developers: Use diffusion-based feature representations trained via 2D supervision to better interpret 3D environments from LiDAR sensor data.
Authors
Samed Doğan, Nico Leuze, Alfred Schöttl
Abstract
Transferring the rich priors of large 2D foundation models to sparse 3D LiDAR remains challenging, as training native 3D foundation models at comparable scale is limited by data and annotation scarcity. We introduce a LiDAR-conditioned diffusion model trained on pseudo-labels from off-the-shelf 2D foundation models. The model supports multiple output modalities, including depth, semantic segmentation and instance prediction, selectable via a textual task prompt. Because the model is conditioned on LiDAR, both its outputs and its intermediate UNet features can be projected back onto the input point cloud, enabling analysis of a 3D representation learned entirely under 2D supervision. We study this representation directly in point-cloud space, explicitly excluding raw spatial coordinates to isolate feature content from projection geometry. Linear probes recover up to ~23% Mean Intersection over Union (MIoU) on 3D semantic classes, compared to ~3.5% for a matched Gaussian-noise control, indicating substantial non-trivial structure. Pairwise cosine similarity across modality-specific feature streams reveals a layered organization. Early encoder layers remain weakly aligned across modalities while individually decodable, intermediate layers converge toward a shared representation, and decoder layers re-specialize toward task-specific outputs. These findings indicate that LiDAR-conditioned diffusion models can induce structured 3D representations from 2D supervision alone, with a modality-dependent manifold that locally unifies near a shared bottleneck. This positions diffusion as a viable mechanism for transferring large-scale 2D priors into sparse 3D domains.