Dexterous robot hands learn to move grasped objects precisely in 3D space
Learning In-Hand Object Reaching to General 6D Poses
Robotics
Summary
Moving objects inside a robot’s hand without letting go is tricky because it requires careful finger movements to change the object’s position and orientation simultaneously. The authors develop a new method called POISE that teaches robot hands through simulation to reach specific 3D positions and rotations of an object held in the palm. Their approach improves success rates by training with varied initial grasps and gradually harder goals, and it works on real robots even when the object is disturbed. This helps robots handle tools or parts more efficiently by adjusting them without dropping or regrasping.
What this means in practice
- •For robotics engineers: Enable robot hands to reposition objects precisely inside their grasp for complex assembly or tool use tasks without dropping.
- •For manufacturing automation teams: Automate parts handling by adjusting object poses within robotic grippers to reduce arm movement and improve efficiency.
Authors
Junxiao Lin, Tianyue Wu, Jie Yin, Jia Pan, Kaifeng Zhang, Weiming Zhi
Abstract
In-hand manipulation allows multi-fingered dexterous hands to reconfigure grasped objects without releasing and regrasping them. This improves manipulation efficiency by reducing repeated grasp acquisition and large arm motions. However, most learning-based methods focus on reorientation, continuous rotation, or translation, whereas many tasks require joint control of object position and orientation. We formulate this capability as in-hand 6D object pose reaching: starting from an existing grasp, coordinated finger motions move the object to a palm-relative target pose. We present POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this task. POISE combines diverse stable-grasp initialization, goal- and geometry-conditioned control, an adaptive 6D goal curriculum, and a compact reward scheme for pose reaching and grasp preservation. In simulation, diverse initialization raises held-out-grasp success from 40.1% to 51.5% and post-drop recovery from 33.8% to 72.9%; the curriculum raises full-range success from 6.2% to 59.5%. On hardware, the grasp-maintenance reward improves three-target sequence success from 20% to 80%. In real-world experiments, POISE reaches successive 6D targets without manual reset across multiple object geometries and wrist orientations, and recovers from external disturbances. To support further research in dexterous manipulation, we will release our code at https://junxiaolin.github.io/poise-website/.