Foreground attraction improves 3d occupancy from camera images
Where Do the Gaussians Go? Foreground Attraction for Camera-Based Weakly Supervised 3D Occupancy
Computer Vision and Pattern RecognitionRobotics
Summary
The paper studies how to better find and represent objects in 3D space using only camera images without detailed 3D labels. The authors focus on a way to update how 3D Gaussians (small blobs representing space) move to better cover foreground objects, which are often missed. They add a new training method that pulls Gaussians toward likely foreground points, improving detection of small objects in tests. This method does not change how the model works when actually used, only during training.
What this means in practice
- •For autonomous vehicle developers: Improve weakly supervised 3D occupancy models used in self-driving cars to better detect small foreground objects from camera images.
- •For robotics engineers: Enhance robot perception systems by increasing accuracy of 3D scene understanding from camera-only inputs without expensive 3D annotations.
Authors
Jeagon Song, Jhonghyun An
Abstract
Camera-based 3D semantic occupancy models can be trained without 3D labels by rendering their predictions into the images and supervising them with 2D pseudo labels. In sparse Gaussian representations trained this way, Gaussian positions are learned only through pixel-wise rendering losses, which we hypothesize give small foreground objects little weight. In trained GaussianFlowOcc models, the Gaussian density on foreground voxels is on average only 1.62 times that on background voxels. We propose a training-only foreground attraction loss that back-projects foreground pseudo-label pixels with pseudo depth and pulls nearby Gaussians toward these 3D points, favoring Gaussians that already predict the target class; the inference network is unchanged. On Occ3D-nuScenes, the loss raises this concentration to 2.40 over three seeds. Over three seeds, the mean foreground IoU increases from 6.29 to 6.96 and is higher for each seed, whereas the change in mean mIoU (13.53 to 13.82) is within the seed-to-seed variation of the baseline. Single-run ablations further indicate that pseudo-label targets outperform targets placed at ground-truth voxels and that acting on the Gaussian positions alone is not sufficient, and a single-seed test shows that the model degrades less when fewer Gaussians are kept at inference.