Image backbone choice strongly affects indoor 3D semantic mapping accuracy
Exploring 2D backbone effects for indoor semantic occupancy prediction
Computer Vision and Pattern Recognition
Summary
Semantic occupancy prediction helps robots and virtual agents understand which parts of a room are empty, occupied, or contain specific objects. The authors show that the image-processing part of this task, called the 2D backbone, has a big impact on accuracy—more than many other parts of the system. By swapping in better image encoders, they significantly improved the 3D understanding without changing other parts. This means the way visual data is processed is crucial for making reliable 3D maps of indoor spaces.
What this means in practice
- •For robotics system developers: Improve the 3D semantic maps used by robots navigating indoors by choosing more effective image encoders for better space understanding.
- •For augmented reality developers: Enhance indoor AR experiences by upgrading image processing backbones to produce more accurate semantic voxel maps of rooms.
Authors
Shizhang Fanga, Wanling Yea, Qi Zheng
Abstract
Semantic occupancy prediction gives an embodied agent a voxel-level account of where space is free, occupied, and semantically meaningful. In RGB-D pipelines such as EmbodiedScan, the image encoder is often left as a default module, even though its features are the visual evidence later sampled into the 3D grid. We study this design choice directly. A central finding is that changing the 2D backbone improves occupancy accuracy more than several carefully designed occupancy architectures or modules. We keep the main RGB-D projection, depth branch, and occupancy head fixed, and replace only the image backbone. The compared encoders are CLIP-ResNet, CLIP-ViT, BLIP2, and DINOv2. Under the controlled setting, the measured mIoU changes substantially: DINOv2 obtains 30.55\%, BLIP2 obtains 29.49\%, CLIP-ViT obtains 24.33\%, and CLIP-ResNet obtains 17.41\%. The stronger encoders also exceed the original EmbodiedScan ResNet-50 baseline without modifying the downstream 3D fusion pipeline. Class-level results give a more detailed picture: DINOv2 is stronger on many layout and structural categories, whereas BLIP2 remains close on several object-centered classes. CLIP-ViT improves clearly over CLIP-ResNet, showing that the way CLIP features are exposed as dense tokens matters for voxel lifting. These results indicate that the image backbone is not a secondary engineering detail in embodied semantic occupancy, but a major source of variation in the final 3D prediction.