Summary
Turning regular video of people moving their upper bodies into robot motions is tricky because human and robot bodies move differently and hands require fine detail. The authors built a system that figures out body and hand movements together from a single camera video and then converts these movements to a robot’s joints while keeping the motion’s geometry intact. They tested their method on videos of people signing and reaching to grab objects, showing improved accuracy in hand positioning and successfully running the robot motions in simulations and on a real robot. This work shows how to go from simple video input to complex upper-body robotic movement in a coordinated way.
What this means in practice
- •For robotics engineers: Convert single-camera human upper-body videos into precise robot arm and hand motions for dexterous manipulation tasks.
- •For sign language technology developers: Create robot demonstrations of sign language gestures from regular videos, improving communication aids with robots.
Authors
Xiaoyu Yang, Sen Han, Da Li, Nan Wu
Abstract
Monocular RGB video provides an accessible source of human demonstrations for upper-body robot motion, yet video-driven human-to-robot transfer remains challenging because body and hand motion are recovered at different spatial scales, human and robot kinematics differ substantially, and fine distal motion is difficult to preserve across embodiments. We present a geometry-preserving motion-retargeting framework that integrates unified body--hand reconstruction with morphology-independent geometric transfer. Frame-wise body estimates, video-level observations, and detailed hand evidence jointly constrain a single differentiable Momentum Human Rig (MHR) state, while transient hand artifacts are repaired in parameter space. The reconstructed motion is represented by arm-segment directions, elbow configuration, relative palm orientation, and bilateral wrist relations, and is realized on the target robot through multi-stage inverse kinematics and robot-specific hand adaptation. Within the broader system, Across-VAM provides video generation, whereas Across-WAM performs human-to-robot motion mapping. The method is evaluated on 16 monocular videos comprising 1,769 source frames, including 10 signing and six reach-to-grasp sequences. Unified reconstruction reduces mean hand reprojection error from 22.36 to 7.21 pixels relative to SAM 3D Body. All 16 retargeted trajectories completed kinematic simulation playback, and representative signing and reach-to-grasp motions were further demonstrated on a physical robot. The results demonstrate a unified pipeline from monocular human video to coordinated upper-body motion on a dual-arm dexterous robot.