Semantic 3D model improves interactive building inspection accuracy
Measurable Semantic 3D Gaussian Splatting with Joint Region and Depth Supervision for Interactive Building Inspection
Computer Vision and Pattern Recognition
Summary
Inspecting buildings accurately requires 3D models that not only look realistic but can measure dimensions and identify parts clearly. This paper presents a new way to build such 3D models using special Gaussian shapes guided by pictures, depth info, and object regions. The method helps create models that can be interacted with in real time, showing both what parts are and their size. The authors also made a platform that lets users explore buildings visually and semantically, helping with tasks like measuring areas and analyzing components.
What this means in practice
- •For building inspection teams: Generate semantically labeled 3D models from images that allow precise measurement and interactive analysis during building inspections.
- •For construction site managers: Use interactive 3D reconstructions for on-site visual and semantic assessment to support maintenance planning and safety checks.
Authors
Yi Zhang, Tian Tan, Ray Kai Leung Su
Abstract
Image-based 3D reconstruction provides a scalable representation for building inspection and maintenance. However, photorealistic reconstruction alone does not fully provide the geometric measurability, semantic structure, and interactive information access required for engineering applications. This study proposes a measurable semantic 3D Gaussian Splatting (3DGS) framework with region and depth supervision for interactive building inspection from multi-view imagery. Mask-based region supervision suppresses irrelevant backgrounds and focuses optimization on target regions. Point-wise depth supervision guides Gaussian centers toward physical surfaces using reference geometry derived from multi-view stereo (MVS) point clouds, providing geometric anchors for measuring physical dimensions and areas. A semantic Gaussian field associates learnable class features with individual Gaussian primitives, enabling semantic rendering and component-level inspection. These capabilities are operationalized through AEC-SplatTwin, an interactive building inspection platform supporting real-time visualization, semantic analysis, quantitative measurement, and multimodal large language model (MLLM)-assisted interaction. On the synthetic UAV-inspection QuakeCity and smartphone-captured HKUPB datasets, the framework reduces Gaussian-center depth MAE by 58.1% and 86.8%, while its semantic field achieves mIoU values of 0.8529 and 0.8642. On HKUPB, the mean relative errors for height, width, and area measurements are 0.33%, 1.07%, and 0.60%. These results show that the proposed framework extends 3DGS beyond visual reconstruction into a measurable, semantically structured, and interactive engineering representation for building inspection.