Robot hands learn to grasp objects for specific tasks from language and vision
OpenDexGrasp: Open-vocabulary Task-Oriented Dexterous Grasping
Robotics
Summary
Grasping objects with robot hands is hard when the robot needs to understand what the task is, like holding a cup to drink rather than just grabbing it. The authors created a system called OpenDexGrasp that combines language instructions, multiple camera views, and detailed object shapes to teach robots how to make hands grasp objects in a way that fits the task. They built a large dataset mixing computer-generated and human-provided grasps to train their approach. Their system can generate robot hand poses that better match the intended function and work well even on objects it hasn't seen before.
What this means in practice
- •For robotics engineers: Program dexterous robot hands to pick and hold objects in ways suited to specific spoken or written instructions.
- •For warehouse automation teams: Implement robot systems that can grasp and manipulate a wide variety of items for packing or sorting based on task descriptions.
Authors
Jiyao Zhang, Junhan Wang, Tianyu Wang, Zeyuan Chen, Anthony Bolten, Yitong Peng, Hao Dong
Abstract
Dexterous grasp synthesis has advanced rapidly in generating stable and physically plausible hand poses, but real-world manipulation requires grasps that preserve the function implied by the task. We study open-vocabulary task-oriented dexterous grasp generation, where a robot must infer functional intent from free-form language, ground it in multi-view visual observations and object geometry, and generate an executable high-degree-of-freedom grasp. We present OpenDexGrasp, a unified data and generative modeling framework for this setting. OpenDexVerse provides dual-source supervision organized by the Coverage-to-Alignment (C2A) Recipe: OpenDex-Scale offers large-scale semantic and geometric coverage through automatic grasp synthesis and vision-language annotation, while OpenDex-Align supplies high-quality embodied alignment through human teleoperation and category-level transfer. OpenDexGrasp learns a shared perception-action latent representation that couples open-vocabulary vision-language context with dexterous action generation. Affordance grounding and grasp generation provide complementary supervision over this latent space, enabling direct generation of task-consistent dexterous grasps without a separate affordance-to-pose inference stage. Extensive simulation and real-robot experiments demonstrate improved functional alignment, physical feasibility, generalization to unseen categories, and real-world execution success. Additional details and videos are available at https://opendexgrasp.github.io/.