Referring objects in dynamic 4D scenes with language and Gaussian splatting
R4DGS: Referring Segmentation in 4D Gaussian Splatting
Computer Vision and Pattern Recognition
Summary
Dynamic 3D scenes that change over time are hard for computers to understand, especially when asked to find specific objects described in natural language. The authors created a method that helps connect language descriptions to objects in these dynamic scenes without needing to train a model on each new scene. They also built a new benchmark to test phrases that refer to objects over time and across complex situations. Their approach uses a special way to represent the 3D scene with Gaussian points and uses language models to break down complex questions into parts that can be tracked and matched to the scene.
What this means in practice
- •For augmented reality developers: Enable natural language queries to identify and track objects in dynamic 3D scenes without scene-specific training.
- •For robotic system engineers: Allow robots to interpret complex natural language commands referring to multiple moving objects in changing environments.
Authors
Bangpu Chen, Yaxuan Li, Shirui Peng, Xiangtian Si, Liuxin Chu, Xitong Cao, Hongbo Jin, Jiayu Ding
Abstract
Dynamic representations based on 4D Gaussian Splatting (4DGS) have shown strong capabilities in reconstruction and rendering, yet current methods remain limited in supporting complex natural-language referring in dynamic scenes. Existing 4D Gaussian methods are primarily designed for rendering-oriented dynamic modeling, while current semantic extensions often rely on scene-specific semantic optimization or are evaluated on relatively simple query settings. In this paper, we study Referring 4D Gaussian Splatting (R4DGS), a task for grounding temporally varying, multi-target or reasoning-intensive, and zero-target or distractor expressions in dynamic 4D scenes. To support this task, we introduce R4D-Bench-QA, a benchmark with structured query annotations. We further present ReferGaussian, a unified framework built upon 4DGS that converts low-level dynamic Gaussians into persistent, language-addressable entities for training-free referring inference. ReferGaussian organizes mask-supported Gaussian entities in an EntityBank, uses Multimodal Large Language Models (MLLMs) to decompose complex queries into trackable object phrases and structured referring constraints, and performs constraint-aware grounding without scene-specific semantic optimization. Experiments on R4D-Bench-QA and the 4D LangSplat protocol demonstrate improved temporal target selection and competitive spatiotemporal grounding without scene-specific semantic training. Project resources are available at https://chenbangpu.github.io/R4DGS/.