Multimodal fusion improves robotic new view synthesis from camera and LiDAR
M3GD: Multi-Modal Multi-View Geometric Diffusion for Camera--LiDAR Novel View Synthesis
RoboticsComputer Vision and Pattern Recognition
Summary
Generating new images of a scene from different viewpoints is a key problem for robots, but existing methods typically use only camera images. The authors show that combining camera images and LiDAR data, which measures 3D points directly, improves the quality of new view images and depth maps. Their method, called M3GD, uses pretrained models for images and point clouds and cleverly aligns their features without extra training. This improves the robot’s understanding of 3D space and lets it better synthesize novel views in real-world settings.
What this means in practice
- •For robotic perception teams: Generate more accurate and metric-consistent novel views combining camera and LiDAR input for navigation and mapping tasks.
- •For autonomous vehicle engineers: Enhance sensor fusion pipelines to improve depth and image synthesis quality for better environmental understanding in self-driving cars.
Authors
Yang Zhou, Jiuhong Xiao, Shizhao Ye, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno
Abstract
Robotic novel view synthesis (NVS) must recover both visual appearance and metric 3D structure, yet most generative NVS methods rely only on images, overlooking LiDAR, a complementary sensor common on robotic platforms. We present M3GD, a Camera--LiDAR multimodal representation for generative NVS that composes independently pretrained 2D image and 3D point-cloud foundation models without separately pretraining a cross-modal translator. We show that, after camera projection, frozen LiDAR and image features exhibit substantial shared spatial structure, providing a natural cross-modal representation. M3GD conditions generation on LiDAR through this structure: it combines explicit geometry statistics with learned point-cloud descriptors into view-aligned packets on the image-latent grid, injected through a lightweight residual adapter into a multi-view flow-matching generator whose latent space, decoders, and training objective remain intact. On the GrandTour dataset, M3GD improves target-view RGB and depth synthesis over an image-only version of the same backbone. Ablations show that the gains come from pixel-aligned LiDAR content and that target-view LiDAR acts as a geometric query linking the requested view to source observations. Deployment on a ground robot demonstrates practical real-world operation, with a configurable quality--cost trade-off controlled by the number of Euler integration steps.