Foundation4D improves multi-camera 3D perception for autonomous vehicles
Foundation4D: Towards a unified perception and reconstruction model for autonomous vehicles
Computer Vision and Pattern Recognition
Summary
Self-driving cars need to understand their surroundings from camera videos to detect objects and estimate geometry. The authors created Foundation4D, a single model that can do many tasks like finding objects, estimating 3D shapes, and figuring out what space is occupied by things. They tested it on several datasets and found it works better than previous methods. Their approach also helps cameras track position more accurately and can support 3D scene reconstruction without lidar sensors.
What this means in practice
- •For autonomous vehicle developers: Build systems that jointly estimate object positions, 3D scene geometry, and semantic occupancy using multi-camera videos to improve perception accuracy.$Commercial implications: Enables companies to offer enhanced multi-camera perception modules for self-driving cars combining detection and reconstruction capabilities.
- •For simulation platform engineers: Use predicted geometry and trajectories without lidar input to generate realistic driving scenes for training and testing autonomous vehicle simulations.
Authors
NVIDIA, :, Sérgio Agostinho, Maximilian Apodaca, Emmanuel Attia, Guillem Braso, George Dong, Ilya Fedorov, Michele Fenzi, Gang Huang, Jiahui Huang, Shengyu Huang, Bingxin Ke, Laura Leal-Taixé, Kuan-Ting Lee, Mingxuan Liu, Tian Liu, Tim Meinhardt, Aljoša Ošep, Naveen Kumar Rai, Kefei Ren, Cristiano Saltori, Keqiang Wu, Ning Xu, Fei Xue, Jin-Bey Yu, Lei Zhang, Qunjie Zhou
Abstract
We introduce Foundation4D, a unified feed-forward model for geometry estimation, 3D detection, semantic occupancy, and Gaussian reconstruction from multi-camera driving video. A shared backbone uses factorized spatiotemporal attention to efficiently combine observations across cameras and time. Our video inference pipeline uses pose graph optimization to refine camera trajectories and assembles object tracks across input windows. We also develop a data annotation pipeline and release PAI-Foundation4D, a benchmark with dense depth and semantic occupancy labels for 1,253 driving clips. With benchmark-specific fine-tuning, Foundation4D achieves the lowest depth AbsRel on Waymo and nuScenes, the highest detection mAP on nuScenes, and the highest occupancy mIoU on OpenOccupancy among the compared methods. Our task-combination study shows benefits for both detection and geometry: co-training detection with geometry and occupancy improves top-class mAP by 1.2 points over detection-only training, while co-training geometry with detection reduces camera-to-rig translation error by $13.4\%$ relative to geometry-only training. We further use the predicted geometry and trajectories as priors for lidarless NuRec, supporting reconstruction of driving scenes for simulation.