Task informed vision language improves 6DoF robotic grasping success
TGrasp6D: Task-Informed Vision-Language Priors and Pose Refinement enable Zero-Shot Task-Oriented 6DoF Grasping
RoboticsArtificial IntelligenceMachine Learning
Summary
Choosing the right way to hold objects is important for robots to complete tasks like slicing bread or handing over a knife safely. Existing methods often focus on simple 2D grasps or need lots of examples for specific tasks. The authors present TGrasp6D, a new approach that uses images and language about the task to decide how to grasp objects in 3D without needing prior examples. Their method improves grasp success significantly and works well on a real robot handling unseen objects.
What this means in practice
- •For robotics engineers: Enable robots to select safe and effective 3D grasps tailored to specific tasks using only a single image and task description without task-specific training data.
- •For automation integrators: Deploy robots capable of handling a variety of objects and tasks in manufacturing or service settings without manual grasp dataset annotation for each task-object pair.$Commercial implications: Allows building adaptable robotic grasping solutions that reduce customization costs for industrial automation providers.
Authors
Manav Kulshrestha, Apoorva Vashisth, Damon Conover, Aniket Bera
Abstract
Task-Oriented Grasping (TOG) has emerged as an important skill for purposeful robotic manipulation. Given the task of using a knife to "slice bread," a robot must choose to grasp from the handle, whereas for "hand[ing] the knife over," it is preferable for the handle to remain unobstructed for safe handoff. Current approaches for TOG either focus on 2D planar grasping, remain limited by their reliance on task-agnostic grasp generators, or are constrained by reliance on finite, manually-annotated, task-specific grasp datasets that cannot cover all possible grasp-task pairs. To that end, we present TGrasp6D, a vision-language based approach for generating task-oriented 6DoF grasps in a zero-shot manner. Given a task and a single RGB-D image of an associated object, TGrasp6D reasons about the appropriate semantic affordance for the given task and object geometry to produce an initial task- informed prior. This generated prior is further refined via 6DoF pose optimization for collision avoidance while preserving task consistency. Our approach achieves state-of-the-art success rate of 94% on the TaskGrasp dataset, with the best baseline at 74%. We also deploy our approach on a Franka Research 3 robot to demonstrate zero-shot generalization to real world objects