Open-vocabulary method improves indoor 3d scene understanding from images
OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation
Computer Vision and Pattern Recognition
Summary
Understanding what things are and where they are in a room from just one photo is hard, especially when the photo shows objects not seen before. The authors developed a new approach that uses language and geometry to help a computer recognize many different kinds of objects and their 3D placement. Their system learns better by combining shape information and word meanings, making it good at guessing unseen objects in indoor scenes. They also created new tests to measure this kind of understanding and showed their method works better than previous ones.
What this means in practice
- •For robotics developers: Create indoor robots that recognize and locate a wider variety of objects from a single camera image, including unseen categories.
- •For augmented reality developers: Build AR systems that map detailed 3D room layouts and identify diverse objects without needing pre-labeled categories.
Authors
Shuyang Yu, Haibo Qin, Shaohan Hu, Wei Shan, Xiaoqing Guan, You Wang, Guang Li
Abstract
Monocular occupancy prediction in indoor scenes aims to recover a scene's 3D geometry and semantics from a single RGB image. Existing methods are restricted to a closed set of predefined categories, limiting their ability to generalize to unseen categories. This limitation is particularly critical in indoor scenes, where objects exhibit highly diverse and fine-grained semantic categories. We study open-vocabulary monocular occupancy prediction for indoor scenes and present OVM-Occ, a Gaussian-based occupancy framework with an instance-aware vision-language teacher. Specifically, we adopt Language-Embedded Gaussians as the intermediate scene representation, propose a geometry-aware deformable attention module to refine Gaussian semantic features, and construct instance-level vision-language embeddings to provide reliable semantic supervision. We further introduce geometry-aware decoupled semantic distillation to mitigate the interference between geometry learning and semantic learning. To evaluate open-vocabulary occupancy prediction performance, we introduce OpenOcc-ScanNet and OpenOcc-Replica, two extended benchmarks with more diverse and fine-grained semantic categories. Experiments show that our method achieves state-of-the-art performance in open-vocabulary monocular occupancy prediction for indoor scenes, outperforming the previous state-of-the-art approach by 12.2 mIoU and 0.6 IoU on the Occ-ScanNet dataset, with consistent gains under new open-vocabulary evaluation protocols.