DIGS improves 3D segmentation with training-free multi-label boundaries
DIGS: Distribution-Informed Gaussian Splatting for Training-Free Open-Vocabulary 3D Segmentation
Computer Vision and Pattern Recognition
Summary
3D segmentation splits a scene into meaningful parts but often struggles at boundaries where one segment blends into another. The authors present DIGS, a new way to represent and identify objects in 3D scenes without lengthy training. Instead of forcing each small part to pick a single label, DIGS keeps multiple label possibilities at boundaries and decides later, leading to cleaner edges. They also combine visual features with language descriptions to better tell apart similar objects quickly and accurately.
What this means in practice
- •For 3d graphics developers: Create precise 3D object masks in scenes without long training, improving interactive rendering applications.
- •For robotics engineers: Enable rapid 3D semantic segmentation to help robots better understand complex environments with minimal setup time.
Authors
Zhifan Zeng, Wenbo Xiao, Arcot Sowmya, Changming Sun
Abstract
Open-vocabulary 3D Gaussian segmentation typically binds semantic features to the per-Gaussian state via per-scene gradient training (1-4 h) or the more recent closed-form distillation. Both commit each primitive to a single feature; therefore, boundary Gaussians spanning multiple 2D instances are forced into a hard decision that degrades retrieval accuracy and yields unstable mask boundaries at render time. We present DIGS, a training-free framework with two stages, class-agnostic mapping and multi-modal encoding, that decouples instance discovery from linguistic naming. At the Gaussian level, our distribution-informed representation maintains an explicit top-$M$ label distribution per primitive, updated by a tempered Dirichlet rule; preservation of multi-peaked label distributions at boundary primitives allows a late-argmax renderer to recover clean instance boundaries. A bidirectional state machine aggregates these distributions into a persistent class-agnostic instance ontology. The instance-level encoding stage then fuses the CLIP visual embedding of each instance's isolated 3D footprint with the CLIP-text embedding of an LLM-generated attribute description, disambiguating visually similar instances. On a single A100, DIGS completes mapping and instance encoding on LERF-OVS in 1.0-3.2 min per scene, depending on whether LLM fusion is enabled, and serves native 3D queries in 5.4 ms. It reaches 71.95% mean mIoU on LERF-OVS, outperforming both the training-free SFS and training-based LaGa baselines. DIGS also achieves the highest mean mIoU on LERF-Mask and 3D-OVS and surpasses Gaga and ObjectGS on Replica under their respective evaluation protocols. Code is available at https://github.com/ZephyrZos/DIGS.