VGOcc: Learning Visual-Geometric Gaussians for Vision-Centric 3D Driving Occupancy Prediction
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed VGOcc, a method that predicts 3D occupancy (which parts of space are filled) using only images from multiple cameras around a vehicle. They use special Gaussian shapes to represent parts of the 3D space but improve on past methods by adding richer geometric and visual information from large pre-trained models. This helps them better place and refine these Gaussians for more accurate 3D scene understanding. Their tests on the nuScenes dataset show that VGOcc performs better than previous vision-only approaches.
3D occupancy predictionGaussian primitivessurround-view camerassemantic 3D occupancydepth ambiguityfoundation modelspose-aware feature learningnuScenes dataset
Authors
Junhong Lin, Xianda Guo, Kangli Wang, Yuqi Ye, Xiaoyu Liang, Yanlun Peng, Wei Gao
Abstract
Vision-only occupancy prediction requires recovering a semantic 3D occupancy field from calibrated surround-view images, where each view provides observations with ambiguous depth along camera rays. Existing methods have progressed from dense structured representations to sparse Gaussian primitives, improving the efficiency of 3D scene representation. However, Gaussian learning still relies primarily on image domain features, which provide limited explicit geometric information for volumetric reasoning. Our key observation is that effective Gaussian occupancy modeling requires not only sparse primitives, but also richer geometric and semantic learning cues. In this paper, we propose VGOcc, which learns visual and geometric cues from foundation models for Gaussian modeling. VGOcc incorporates these cues into primitive initialization and refinement, yielding a representation termed Visual-Geometric Gaussians tailored to semantic occupancy prediction. Specifically, we propose Visual-Geometric Gaussian Birth to form spatially balanced Gaussian centers from ray depth hypotheses, while visual semantic features initialize primitive attributes. Next, we design Pose-Aware Feature Learning to combine foundation tokens with camera embeddings and calibrated ray information. Features from neighboring views are then aggregated at projected 3D locations for each Gaussian refinement stage. Finally, Gaussian decoder refines birth Gaussians with pose-aware features and renders them into semantic occupancy. Experiments on nuScenes demonstrate that VGOcc achieves state-of-the-art performance in vision-only 3D occupancy prediction. Codes will be available at https://github.com/JHLin42in/VGOcc.