ViT-H enables direct video-to-humanoid control with improved motion tracking
ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback
RoboticsComputer Vision and Pattern Recognition
Summary
Controlling humanoid robots using regular human videos is challenging because existing methods break the process into separate steps that don't always work well together. The authors present ViT-H, which can directly create robot motions from videos without needing extra steps like reconstructing the human motion. ViT-H also learns from how well the robot can follow these motions, making the result more accurate and easier for robots to perform. This system can even control humanoid robots in near real-time with very low delay.
What this means in practice
- •For robotics engineers: Create humanoid robot controllers that generate motions directly from video recordings for improved execution quality.
- •For robotics teleoperation teams: Perform real-time teleoperation of humanoid robots controlled by live human video input with sub-400 ms latency.
Authors
Zimo He, Tianyi Bian, Zizhong Yang, Hongjie Li, Nan Jiang, Jieming Cui, Baoxiong Jia, Yizhou Wang, Siyuan Huang, Yixin Zhu
Abstract
Learning humanoid control from monocular human videos offers a scalable way to acquire diverse whole-body behaviors. Yet, existing reconstruction-retargeting-tracking pipelines are limited in system-level scalability by separately optimized intermediate stages and cannot directly optimize reference motion executability using downstream tracking feedback. We present ViT-H, a video-to-humanoid control framework following a generation-and-tracking paradigm. ViT-H directly generates humanoid reference motions from videos with a diffusion model, bypassing human motion reconstruction and retargeting, and uses motion-tracking outcomes to post-train the generator with direct preference optimization. We further introduce an autoregressive variant that enables real-time humanoid video teleoperation with sub-400 ms latency. Extensive experiments on the Unitree G1 demonstrate that ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines in both motion quality and executability. We further show that ViT-H scales favorably with model size and with both pre-training and post-training data scale, while motion-tracking feedback consistently improves executability across different motion trackers.