Vision language models improve object segmentation and depth for robots
VLMs Can Describe, But Not Measure: Object-Centric Scene Understanding for Robotic Manipulation
RoboticsComputer Vision and Pattern Recognition
Summary
Robots need to both recognize objects and know exactly where they are to work well in new places. The authors show that using vision-language models (VLMs) alone isn’t enough for measuring object positions accurately. They combined VLMs with depth sensing to better separate and locate objects from a single view. This approach helps robots understand scenes more precisely and can be used for planning tasks like picking up items.
What this means in practice
- •For robotics engineers: Create robot perception systems that combine object labeling with accurate position and depth data for better manipulation in unknown environments.
- •For automation integrators: Integrate object-centric semantic and depth information to improve robotic task execution on varied tabletop scenes for manufacturing or logistics automation.
Authors
Enrico Saccon, Tommaso Faraci, Iñigo De La Ossa Zarzuelo, Luigi Palopoli, Marco Roveri, Matteo Saveriano
Abstract
Robotic operation in previously unseen environments requires both semantic understanding and reliable metric information. While vision--language models (VLMs) provide strong semantic capabilities, their geometric estimates remain less reliable. In this paper, we propose a VLM-driven, modular perception framework for scene understanding using off-the-shelf approaches. Starting from a single RGB-D observation, the scene is segmented into object-level regions, annotated by a VLM, and grounded with depth information to construct a task-independent object-centric representation. Experiments on 151 tabletop scenes show that the proposed decomposition preserves strong semantic performance while substantially improving localization and depth estimation over direct VLM inference. The resulting representation is also integrated with a task-planning framework for robotic execution.