Construction robots improve partly hidden object detection using geometry features
Cut a Wall, See a Wall: Identity Substitution in Occluded Point-Cloud Perception for Construction Robots
Robotics
Summary
Robots working on building sites need to identify parts of structures even when they can only see parts of them. The authors found that if a robot’s vision is blocked by large objects it wasn’t trained to handle this way, it often mistakes one object for another instead of just missing it. They improved the robot’s recognition by adding specific geometric details about shapes, which helped the robot correctly identify objects even when parts were hidden. This approach worked better than simply training the robot with more blocked views and improved performance on both synthetic and real building scans.
What this means in practice
- •For construction robotics teams: Enable robots to better label partially hidden building parts to improve automated inspection accuracy on real and simulated construction sites.
- •For 3d scene understanding developers: Incorporate geometric point features into segmentation models to reduce object misclassification caused by occlusions in 3D point clouds.
Authors
Tsung-Lung Yang
Abstract
Construction robots must label building elements they can only partly see. We show that when occlusion at test time removes whole regions of a scene, rather than parts of an object as in training, point-cloud segmentation networks do not simply miss the hidden elements: they relabel them, and the errors collapse onto a single sink class. With fixed synthetic layouts the column absorbs up to 70% of all test points; with randomized layouts the sink moves to the stair, predicted at 4.5-5.4x its true share across three training seeds. Baseline mIoU falls to 0.15-0.27 (fixed) and 0.36-0.72 (randomized), and up to 269 whole objects vanish in a single test set. Appending 14 hand-computed geometric features per point (GPE) restores mIoU to 0.54-0.92 and removes 64-100% of whole-object losses in every run, for PointNet++ and a PTv3-style transformer in both layout regimes; on real S3DIS scans it removes every whole-wall loss (10 to 0). In an as-built column check computed from the predictions, GPE raises the share of correctly checked views from 17-58% to 71-92%. Feature permutation on fixed, randomized and real scenes explains the pattern: each class relies on whichever cue is stable in its training data, position for height-banded slabs and beams, intrinsic shape for walls and stairs once their position varies. Occlusion augmentation is a strong alternative on mIoU, but in every setting it loses more whole objects than GPE (e.g., 25-35 vs. 0).