OmniHOI enables robot hands to imitate human hand-object video
OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video
Robotics
Summary
It’s hard for robots to learn how to handle objects by watching regular videos of people using their hands, because videos only show one angle and aren’t precise. The authors created OmniHOI, a method that turns a single video of a person manipulating an object into detailed movements that robot hands can follow. Their method checks physical rules like contact and motion during different steps to make sure the robot’s hand movements are accurate and realistic. This results in better success rates when robots mimic human object interactions, working well even on real robots.
What this means in practice
- •For robotics engineers: Generate realistic dexterous hand manipulation trajectories from single-view videos for use in robot control.
- •For robotics integrators: Enable bimanual robots to perform diverse object manipulation tasks by transferring human hand-object interaction from standard videos.$Commercial implications: Allows creating advanced robot manipulation products that can learn skills from ordinary human videos, improving automation capabilities.
Authors
Ting Mao, Yanming Shao, Ziheng Wang, Haoyu Liu, Yiqun Wang, Xuanye Wu, Yao Mu
Abstract
Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-in-the-loop refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39-89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.