Hand movements teach robots complex object handling moves
CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements
RoboticsComputer Vision and Pattern Recognition
Summary
Robots often struggle to imitate complicated hand movements needed to handle objects that involve rotations or flips. The authors collected a large set of paired examples where human hands move objects, and robots mimic these actions using grippers. They developed a two-step learning process that first predicts key robot poses, then fills in the full motion sequence, improving accuracy. This approach was tested both in simulations and on real robots, showing better performance than previous ways of copying hand motions.
What this means in practice
- •For robotics engineers: Use paired hand-to-gripper data and a two-stage method to teach robots complex object manipulations involving rotations and flips.
- •For industrial automation teams: Improve robotic assembly lines by enabling robots to perform intricate hand-like movements for handling diverse objects.
Authors
Hongxiang Zhao, Mutian Xu, Zeyu Jin, Yiming Hao, Shuguang Cui, Xiaoguang Han
Abstract
Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot learning. However, existing methods are largely confined to simple, planar tasks and fail to handle complex spatial movements (e.g., intricate trajectories involving rotations or flips) that are essential for robot manipulation. Motivated by this gap, we adopt an implicit, data-driven approach guided by fine-grained hand-pose motions. To this end, we introduce a scalable acquisition pipeline to collect hand-gripper paired demonstrations, governed by a rigorous protocol that prioritizes motion complexity and leverages a handheld gripper for seamless action mimicry. This yields a large-scale paired dataset comprising 6,189 episodes across 1,254 unique objects, exhibiting significantly higher spatial complexity than existing benchmarks. However, learning such complex mappings remains challenging. We observe that naive end-to-end generation of full gripper pose sequences is insufficient, as minor trajectory deviations compound rapidly under intricate dynamics. To address this, we propose a two-stage framework: Stage I predicts sparse gripper keyframes (initial and terminal) to simplify the mapping objective, while Stage II generates the full continuous action sequence conditioned on these keyframes. Furthermore, to mitigate cumulative drift, we keep the gripper's orientation being learned while post-optimizing its translation based on the grasping heuristic and kinematic consistency. In both simulation and real-robot experiments, our framework enables stable and precise hand-to-gripper transfer of complex spatial manipulations, significantly outperforming traditional baselines. Project page: https://cosmoh2g.github.io.