Trackgraph builds fast, flexible 3d maps with natural language labels
TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking
Computer Vision and Pattern RecognitionRobotics
Summary
Mapping unknown environments so robots can understand and describe them in natural language is a challenge because existing methods are slow and use a lot of computing power. The authors developed TRACKGRAPH, a system that tracks 2D shapes in a video stream and fuses them into 3D segments efficiently, reducing the need for repeated heavy computations. This approach makes real-time, open-vocabulary 3D scene graphs possible, meaning robots can recognize and find objects described in many different words. They showed that TRACKGRAPH is faster and uses less memory than previous methods while working well on several standard datasets and real robots.
What this means in practice
- •For robotics developers: Construct open-ended 3D scene maps onboard robots that label objects flexibly using natural language during exploration.
- •For drone operators: Create detailed 3D environment models from aerial video with searchable object labels for inspection or monitoring tasks.
- •For augmented reality developers: Enable AR systems to identify and track diverse objects in 3D scenes with natural language queries using efficient online mapping.$Commercial implications: Allows building new AR products offering dynamic scene understanding with flexible object search beyond preset categories.
Authors
Peder Borge Hellesylt, Albert Gassol Puigjaner, Kostas Alexis, Annette Stahl
Abstract
Open-vocabulary 3D maps enable robots to reason about previously unknown environments using natural language. However, existing systems typically segment every incoming image, associate detections with persistent 3D segments, and frequently perform costly Vision-Language (VL) inference. We present TRACKGRAPH, an online open-vocabulary system that maintains short-term 2D mask identity directly in the image stream before fusing segments into 3D. FastSAM masks and CLIP features are computed at sparse keyframes, while dense DINOv3 features are used to propagate masks at a high rate in between. The resulting tracked masks are fused into a class-agnostic 3D segment layer within a hierarchical scene graph, with 3D association handling tracking interruptions and long-term revisits. Compact multi-view CLIP embeddings enable open-vocabulary retrieval. Across Replica, ScanNet++, and HM3D, TRACKGRAPH achieves competitive open-vocabulary segmentation and retrieval against state-of-the-art mapping methods, including the highest synonym frequency on Replica (0.50). On the same NVIDIA A100, it is 1.7x faster and uses 3.3x less GPU memory than ViT-H OVI-MAP. Real-world quadruped deployments demonstrate onboard scene graph construction and object search at 7.5Hz, while recorded drone data is used to test the method under aerial viewpoints.