Hand skeleton model helps robots learn tasks from humans more effectively
Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer
Robotics
Summary
Teaching robots to do tasks can be slow and expensive because it’s hard to gather many examples of robot actions. The authors found a way to help robots learn from videos of humans by focusing on the motion of hands using a shared hand skeleton model. This approach helps the robot understand and imitate human hand movements better, even when the robot’s hands look different from humans. Their method improved robot success in manipulating objects across several tasks, especially when learning from both human videos and robot demonstrations.
What this means in practice
- •For industrial robot programmers: Improve robot training workflows by integrating human video demonstrations to expand task variations without needing robot-specific labels.
- •For robotics simulation engineers: Develop simulation environments that combine human and robot motion data to create more robust bimanual manipulation models.
Authors
Zetao Cai, Yaping Li, Yiqun Wang, Xinyu Zhan, Yuyin Yang, Haoxiang Ma, Kailin Li, Tao Lu, Jiangmiao Pang, Linning Xu, Dahua Lin
Abstract
Robot demonstrations are expensive to collect and often provide limited distributional coverage of task variations. Human videos offer a low-cost source of complementary manipulation experience, but learning from them requires bridging embodiment gaps in visual appearance and action spaces. We introduce Skel-WAM, a world action model that bridges these differences through a unified hand-skeleton motion interface. The key insight is to align human and robot motion through a common hand topology, combining skeleton overlays that ground motion in the scene with structured 2.5-D keypoints that encode explicit hand kinematics. Video and Keypoint Experts jointly learn visual and skeletal dynamics through a Mixture-of-Transformers, while a separate robot-trained Action Expert maps these predictions to executable controls. This separation enables human and robot demonstrations to directly supervise shared dynamics without requiring robot action labels for human videos. Across four real-world bimanual tasks and seven simulated tasks, Skel-WAM achieves average success rates of 79.86% and 63.29%, surpassing the strongest baseline by 22.22 and 8.28 percentage points, respectively. Human-robot cotraining more than doubles real-world success on task variations absent from robot training data, from 38.89% to 86.11%. These results demonstrate that a shared skeletal interface enables joint learning across human and robot data and expands robot task coverage through complementary human demonstrations.