Semantic-augmented experts improve point cloud alignment in low-overlap scenes
SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration
Computer Vision and Pattern Recognition
Summary
Aligning 3D scans of scenes can be tricky when only small parts overlap and parts look alike but are actually different objects. The researchers created a method called SAGE that adds knowledge about what the objects are (semantics) to help match points better. It uses a special way to combine experts that focus on specific object types and shapes without changing the original object recognizer. This makes the alignment more accurate, especially when there’s little overlap between scans.
What this means in practice
- •For 3d mapping engineers: Improve the accuracy of 3D map creation from partial scans where overlapping areas are small or ambiguous.
- •For robotics navigation teams: Enhance robot localization by better aligning sensor scans in environments with visually similar but semantically different objects.
Authors
Tomoyasu Shimada, Kazuhiko Murasaki, Shoichiro Takeda, Shogo Sato, Riku Inoue, Toshihiko Nishimura, Ryuichi Tanida
Abstract
Point cloud registration is an essential task for scene reconstruction, requiring the alignment of partially overlapping point clouds. Under low-overlap conditions, this task remains challenging because existing geometry-only methods cannot distinguish geometrically similar but semantically distinct structures (e.g., TV corners vs. desk corners and chair legs vs. table legs), resulting in incorrect correspondences. To address this geometric ambiguity, we present Semantic-Augmented Geometric Experts (SAGE), which integrates frozen pre-trained point cloud semantic representations into the registration pipeline without requiring additional sensors or fine-tuning the semantic encoder. To effectively leverage these semantic distinctions, SAGE employs a Semantic-Guided Mixture of Experts (SG-MoE) architecture. Through semantic-aware routing, SG-MoE enables each expert to specialize in distinct semantic-geometric patterns, reducing mismatches that confound geometry-only methods. With a structure-aware estimator, SAGE achieves the highest registration recall on 3DLoMatch while remaining competitive on 3DMatch. When evaluated with the same number of refinement iterations, SAGE further achieves the highest inlier ratio and registration recall on both benchmarks. These results indicate that frozen semantic representations provide effective cues for resolving geometric ambiguity in low-overlap registration.