LiFi generates accurate LiDAR scenes from multiple camera views
LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating 3D LiDAR images from regular camera photos is useful but challenging. Existing methods often use just one camera or focus only on the picture's content, which limits accuracy. The authors developed LiFi, a method that combines the shape details and the meaning from several camera views to create better 3D LiDAR scenes. Their approach balances geometric and semantic information to improve precision and quality. Tests show LiFi works better than current methods for making detailed LiDAR data.
What this means in practice
- •For autonomous vehicle teams: Generate high-quality LiDAR data from multi-camera setups to improve vehicle perception without needing physical LiDAR sensors.
- •For simulation engineers: Create realistic LiDAR scenes from images for cross-modal simulation environments, enabling testing of perception systems with generated sensor data.
Authors
Sizhuo Zhou, Xiaosong Jia, Fanrui Zhang, Qifeng Li, Junjie Li, Zirui Wang, Yukang Feng, Yu Hong, Shaofeng Zhang, Wenlong Liao, Tao He, Juyong Zhang, Junchi Yan
Abstract
Generating LiDAR point clouds from multi-view camera images is a valuable task with applications in controllable data synthesis, cross-modal simulation and various perception tasks. However, most existing methods are not specifically designed for this task, and the quality of their generated LiDAR remains limited. Moreover, these methods typically focus only on semantic cues in images or generate LiDAR from a monocular camera image. In this work, we introduce LiFi (LiDAR From images), a dedicated framework for generating visually and geometrically aligned LiDAR scenes from multi-view camera images. LiFi guides the latent diffusion process through two complementary branches: geometry and semantics. The geometry branch constructs scene features from images through a reverse sampling algorithm and an uncertainty-aware geometric encoder combined with DepthAnything3, whereas the semantic branch provides high-level semantic representations via a cross-view interaction module. We further introduce a dual-branch balanced classifier-free guidance (CFG) strategy, which enhances the model's conditional generation capability while preserving the independent learning and collaborative controllability of the two branches. Extensive experiments demonstrate that LiFi outperforms state-of-the-art methods in generating high-fidelity LiDAR scenes. The code will be publicly available at: https://github.com/szzhou88/LiFi.