Visual transformers segment 3D object relations without pose input
RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding how objects relate to each other in 3D scenes is important but hard when you don't know exactly where the cameras are or the objects' names. The authors created RelationVGGT, a computer method that looks at multiple images and a text description to find the related object, even without knowing its category or camera poses. It uses advanced visual and 3D geometry tools combined with a transformer model to predict these relationships directly and quickly. They also made an automated way to create training data using existing 3D scene collections combined with AI language and vision models.
What this means in practice
- •For augmented reality developers: Segment related 3D objects across camera views using text queries without requiring camera positions or object categories.
- •For robotics system integrators: Enable robots to identify and segment objects based on spatial relations from multiple viewpoints without needing exact camera poses.
Authors
Minsu Kim, Jaesung Choe, Jiwoo Lee, Yu-Chiang Frank Wang, Seon Joo Kim
Abstract
Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction -- requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.