Humanoid robots learn motions directly from single camera videos
BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video
RoboticsComputer Vision and Pattern Recognition
Summary
Teaching robots to move like humans is hard because their bodies are different and usual methods rely on first understanding human movement. The authors created a new way that lets robots learn motions straight from regular videos without needing a detailed human model. Their method also helps make the robot's movements smoother and more natural by considering how the robot touches the ground. This approach works better and faster on both computer tests and real robots.
What this means in practice
- •For robotics engineers: Create robot motions directly from human video without needing an explicit human motion model.
- •For robotics software developers: Improve humanoid robot motion execution by integrating contact-aware optimization for smoother and physically plausible movements.
Authors
Tianyu Xiong, Yi Lu, Jinrui Wang, Ziqi Liang, Dandan Lei, Xiaoyang Zhou, Xiao-xiao Long, Qiu Shen, Xun Cao
Abstract
Learning executable motions from human videos offers a scalable solution for humanoid robots to acquire demonstration motions. However, existing pipelines typically first construct an explicit human motion representation and then convert it into robot motions via motion retargeting. Although such methods can effectively leverage large volumes of existing human data for training, the substantial differences between humans and humanoid robots in locomotion mechanisms and joint degree-of-freedom configurations make motions generated by this human-representation-centric approach difficult to execute on robots. Furthermore, errors introduced during human motion estimation inevitably propagate to the retargeting stage and cannot be eliminated via joint optimization. We propose BeyondRetarget, an end-to-end framework that directly maps monocular RGB videos to robot motions. Discarding the explicit human representation, this framework learns robot-oriented implicit representations directly from visual observations, enabling the model to capture cross-morphology motion structures. To generate motions more suitable for robot execution, we further design a contact-aware motion optimization mechanism to improve temporal consistency and physical plausibility. Experiments show that BeyondRetarget significantly improves the accuracy and robustness of generated robot motions, while achieving higher execution success rates and lower latency in both simulation environments and real humanoid robots.