FOUND-IT: Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand
2026-05-25 • Robotics
Robotics
AI summaryⓘ
The authors developed a new method, FOUND-IT, that creates detailed 3D maps of indoor or outdoor places using just a regular camera in real-time. Their system can change how detailed the map is based on what the robot needs to do, like focusing on small objects for grabbing or big objects for moving around. Unlike previous work, their method adapts as new tasks come up, allowing robots to handle complex activities more flexibly. They showed that FOUND-IT works well on benchmarks and real robots, even using casual videos from the web.
3D scene graphmonocular camerageometric foundation modelstask-driven mappinglocomotionmanipulationreal-time processingautonomous robotsenvironment representationVGGT
Authors
Dominic Maggio, Nicolas Gorlo, Luca Carlone
Abstract
We present the first approach to build hierarchical task-driven 3D scene graphs of arbitrary indoor or outdoor environments using an uncalibrated monocular camera in real-time. We leverage geometric foundation models to estimate geometric attributes of the scene graph (e.g., object bounding boxes), but we also observe that traversability information (the "places" layer of a scene graph) can be directly reconstructed by adding an extra head to existing geometric foundation models, like VGGT. Our approach is task-driven in the sense that we adjust the granularity of the objects and regions in the map depending on the task; for instance, during a manipulation task, our approach is able to resolve small knobs on a stove, while during a navigation task it can focus on large objects (e.g., the entire stove). However, in a major departure from related work, we consider the realistic case where the list of tasks is not predefined and fixed, but evolves as the robot operates. This naturally allows dealing with complex loco-manipulation tasks, where the robot can dynamically adjust its representation as the task unfolds. We dub the resulting approach FOUND-IT. FOUND-IT also includes an agentic approach to query information in the scene graph. In addition to achieving 79% higher accuracy on the ASHiTA SG3D task grounding benchmark, we demonstrate FOUND-IT runs in real-time on a ground robot using a Jetson Thor. Furthermore, to highlight the robustness of our method, we demonstrate constructing 3D scene graphs on casually captured realtor apartment tours from YouTube. Code will be made available upon publication.