Dexterous robotic hand reaches objects safely with wrist camera
Pushing Dexterity and Safety for In-Hand Object Pose Reaching That Works Remarkably Well
RoboticsMachine Learning
Summary
Robotic hands that can skillfully move and reorient objects inside their grip have been hard to train and keep safe. The authors created a training approach that lets a multi-fingered robot hand move objects precisely to many different positions and angles without dropping them. The system uses simulation data, adapts friction realistically, and learns to balance performance with safety via a safety filter. It works well on a real robot using just a small camera by the wrist, even for tricky tasks like spinning a lightbulb.
What this means in practice
- •For robotics engineers: Control multi-fingered robotic hands to manipulate and reposition objects safely across diverse tasks without dropping them.
- •For industrial automation teams: Use wrist-mounted cameras and learned policies to install or assemble objects in confined spaces, such as screwing in lightbulbs.$Commercial implications: Enables reliable dexterous manipulation in automation products for maintenance and assembly, improving robot functionality in real-world tasks.
Authors
Ian Chuang, Hiya Shah, Vamsi Chilamkur, Iman Soltani, Claire Tomlin
Abstract
Dexterous manipulation with multi-fingered hands is the next frontier of robotics, yet learned hand skills remain limited to simple grasps or basic repeated motions, such as rotating an object. As robots attempt more demanding in-hand maneuvers, safety, such as not dropping the object, becomes critical, yet it is rarely measured. We study in-hand object pose reaching, a task that demands diverse and difficult grasps: moving an object to almost any position and orientation within reach of the fingers, at any wrist orientation, through long sequences of consecutive goals. We present a recipe for training such policies for a LEAP hand in simulation without human data, built on a range of insights, from diverse resets and goals via generated grasp datasets to an adaptive friction curriculum. With careful system identification, the policies transfer zero-shot to the real hand, using either motion capture or a single wrist-mounted depth camera, and prove remarkably capable and robust. On top of the task policy, we add a learned residual safety filter with adjustable conservativeness, letting us tune the balance between performance and safety and greatly reduce object drops. In 13.3 hours of real-robot evaluation over 4,800 unseen goals across 4 objects and diverse wrist orientations, the state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad, and the filter cuts its drops by 75% while still reaching 91.5% of goals. The policies can also spin a lightbulb held in an arbitrary pose, reach out-of-distribution goals, withstand fast arm motion and perturbations, and, needing only a wrist camera, serve as portable skills such as screwing in a lightbulb. We release the full recipe as a platform for studying performance, reliability, and safety in high-dimensional dexterous systems. Videos and more are available at https://ian-chuang.github.io/dexsafe/.