Model estimates robotic object joint positions from a single image
Query-Conditioned Articulation Estimation from a Single Image
Robotics
Summary
Robots often need to understand how parts of objects like doors or drawers move so they can interact with them better. This work presents a new method called QueryArt that lets a robot figure out an object's movable parts and their positions by looking at just one photo, even if the object is new to the robot. Instead of trying to identify and segment parts first, QueryArt uses a special point the robot asks about and works out the motion relative to that point, which helps improve accuracy. The authors show that QueryArt performs better than existing methods and works well even on real robots testing many different objects.
What this means in practice
- •For robotics engineers: Enable robots to identify parts and motion of unknown objects from a single image to improve manipulation tasks.
- •For robotic system integrators: Integrate visual articulation estimation for mobile robots to enhance success rate in varied real-world object interaction.
Authors
Abdelrhman Werby, Fabio Scaparro, Kai O. Arra
Abstract
Enabling robots to estimate the kinematic parameters of articulated objects unlocks a wide range of capabilities for interaction and manipulation. The estimation has to happen from the information the robot currently observes, often just a single RGB image of an object it has never seen before. Current single-image approaches couple articulation part segmentation with articulation estimation, making their predictions vulnerable to missed detections and incorrect part associations, and they regress metric 3D geometry that a single view fixes only up to scale. We present QueryArt, a model that estimates articulation parameters from a single RGB image, a 2D query point, and camera intrinsics. QueryArt is trained to estimate the 3D articulation geometry relative to the queried point and in units of its depth, which keeps its target identifiable from the image alone. A single depth measurement at the query point then supplies the scale and recovers the metric parameters. We train QueryArt on a curated mixture of synthetic and real-world articulation datasets. We evaluate QueryArt on several benchmarks and compare it against recent baselines. QueryArt outperforms recent baselines on most articulation metrics, including on out-of-distribution data. To demonstrate the model's capabilities in real-world settings, we evaluate QueryArt on a mobile manipulator across 57 manipulation trials spanning 16 object parts and five viewpoint classes, achieving a 70.2% success rate. We provide code and videos at: https://abwerby.github.io/queryart/