ClusterSplat improves 3D object identification from natural language in complex scenes
ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting
Computer Vision and Pattern Recognition
Summary
This paper tackles the challenge of finding specific objects in 3D scenes based on natural language descriptions. Previous techniques picked target objects by using simple scoring rules, which could be unstable because scores vary a lot depending on the scene or description. The authors propose ClusterSplat, which groups parts of the 3D scene into meaningful clusters adapted to each scene, then picks the best cluster that matches the description. This method works better at both marking the object in 2D views and identifying it in 3D space across various datasets.
What this means in practice
- •For augmented reality developers: Accurately select and highlight objects in 3D scenes using spoken or written descriptions for improved interactive AR experiences.
- •For robotic navigation teams: Identify target objects in complex environments from verbal commands by detecting 3D clusters that correspond to descriptions.
Authors
Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung Kim
Abstract
We study 3D visual grounding in 3D Gaussian Splatting (3DGS), where a referring expression should identify a target object as both a rendered 2D mask and a set of explicit 3D Gaussians. Prior methods typically extract 3D targets by applying heuristic thresholding or top-ratio filtering to dense primitive-level relevance scores. However, these scores often vary substantially across scenes and prompts, making such heuristics sensitive and leading to unstable targeting. ClusterSplat addresses this by reframing 3D target selection as cluster-level selection over scene-adaptive candidates. The method first learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seed voxels when a description-length criterion decreases, producing a scene-specific set of cluster candidates. Given a referring expression, a query-conditioned cluster scorer ranks the cluster candidates with a lightweight MLP, and the selected cluster directly defines the explicit 3D target while its cluster score is rendered for the 2D mask. On ScanRefer, ClusterSplat achieves the best rendered 2D segmentation and explicit 3D target metrics among the compared 3DGS baselines. It also obtains the best average Ref-LERF scores, supporting the same trend under fine-grained referring descriptions. On ScanNet v2 class name queries, ClusterSplat achieves the strongest rendered 2D results and the highest 3D matching accuracy, while maintaining competitive 3D IoU. Our project page is available at https://jaesung8.github.io/clustersplat/ .