Gaussian neural fields improve camera based occupancy prediction
GNFormer: Gaussian Neural Field Transformer for Efficient Camera-based Occupancy Prediction
RoboticsComputer Vision and Pattern Recognition
Summary
Predicting what parts of a 3D scene are occupied using only camera images is hard and useful for robots and self-driving vehicles. The authors introduce a new way to represent space using a combination of simple 3D shapes called Gaussians plus a neural network to better predict which areas contain objects. This approach uses fewer Gaussian shapes and runs faster while giving more accurate results than previous methods. Their method works well both indoors and outdoors and can be used in real-time for robotic perception.
What this means in practice
- •For autonomous vehicle developers: Enhance real-time 3D scene understanding from cameras to improve navigation and obstacle detection for self-driving cars.
- •For drone system engineers: Provide efficient, accurate environment mapping from cameras for UAVs in both indoor and outdoor settings.
Authors
Guangcheng Chen, Li He, Hong Zhang
Abstract
Camera-based occupancy prediction provides geometric and semantic scene understanding for robot perception and has been adopted in autonomous vehicle and UAV perception systems. Gaussian primitives offer a promising alternative by potentially representing only occupied regions, thereby concentrating computation and representational capacity where needed and reducing wasted computation and memory in empty space. However, existing Gaussian-based methods suffer from poor primitive utilization, as ineffective training objectives leave some Gaussians in empty space, while the limited expressiveness of Gaussian functions restricts their ability to model complex shapes. To address these limitations, we propose a novel 3D representation, Gaussian Neural Field (GNF), which can accurately map voxel positions to semantic occupancies. In GNF, Gaussians and a neural network jointly parameterize the field, where the Gaussians define its spatial support and provide features for neural decoding, while the neural network enables more expressive occupancy modeling than Gaussian functions alone. We further develop the Gaussian Neural Field Transformer (GNFormer), a streamlined and flexible method that predicts GNFs for camera-based occupancy prediction. Across Occ-ScanNet, SSCBench-KITTI-360, and nuScenes-SurroundOcc benchmarks, GNFormer achieves the highest IoU and mIoU and the fastest inference among the compared single-frame methods. In the experiments, GNFormer can represent indoor and outdoor scenes with only 256 and 1024 Gaussians, respectively, substantially fewer primitives than prior Gaussian-based methods. The code will be released.