Fine tuning SAM 2.1 improves building component segmentation in 3D images
Domain Adaptation of SAM 2.1 for Building-Component Segmentation in Image-Based 3D Reconstruction
Computer Vision and Pattern Recognition
Summary
Segmenting parts of buildings in photos helps create accurate 3D models from those images. The authors fine-tuned a powerful image segmentation model called SAM 2.1 using a dataset of labeled building photos to better identify building parts like walls and windows. This improved the accuracy of segmenting these parts by about 20 percent compared to the original model. Their improved model also performed better than a baseline method on larger building components. This helps make 3D reconstruction methods more precise when grouping these parts together.
What this means in practice
- •For architectural visualization teams: Generate more accurate 3D building models by using the fine-tuned segmentation model to better identify structural components in image-based scans.
- •For construction project managers: Improve progress tracking and site analysis by using enhanced 3D reconstructions with more precise building component segmentation.
Authors
Ehsan Agha Ebrahimi, Yong-Cheol Lee, Keith G. Mills
Abstract
3D reconstruction of building components using image-based scanning requires appropriate segmentation of the objects in the 2D building images. In order to benefit from consistent grouping of the 2D masks in 3D environments, vision foundation models such as the Segment Anything Model have been utilized by Group Anything with Radiance Fields (GARField) to perform 3D segmentation. This study fine-tuned SAM 2.1 on the HBD dataset, consisting of 2,679 annotated images from 42 buildings, to enhance multi-class segmentation of 2D building images, covering 11 classes (Wall, Window, Ceiling, Door, Column, Beam, Floor, Opening, Curtain Wall, Roof, and Lift). The model was evaluated using per-class IoU, Dice, and mIoU metrics, and the results show an improvement in mIoU from 49.9% to 69.6%, indicating a meaningful improvement in class segmentation and providing a more accurate model for distilling the masks in 3D. SegFormer was then used as a baseline, generating per-class masks for comparison with SAM 2.1, which indicates improvement in some object classes with larger surface area. The masks in the building images were grouped in the neural radiance field, and the pretrained SAM 2.1 was compared with the fine-tuned SAM 2.1 in the newly rendered images. This comparison shows that fine-tuning improves the grouping of building components at a user-selected grouping scale. This advances the application of Group Anything with Radiance Fields (GARField) in 3D, enabling better segmentation of building structures for 3D auto-reconstruction.