Simulation-trained humanoid robots succeed more without real data
Zero-HLM: Generalizable Humanoid Loco-Manipulation without Real Robot Data
Robotics
Summary
Training humanoid robots to perform tasks that involve moving and manipulating objects usually needs lots of real-world examples, which are expensive to gather. The authors created a way to use smartphone videos and computer simulations to make many training examples instead. Their approach lets robots learn to do three tasks well without needing real robot data for training, even when the task setups change. Robots trained this way succeeded more often than those trained with limited real examples.
What this means in practice
- •For robotics engineers: Create more adaptable humanoid robot control systems using scalable simulated data instead of costly real-world demonstrations.
- •For automation system integrators: Deploy humanoid robots in complex environments by training them in simulation for better zero-shot task success without requiring new real robot data.
Authors
Dianyi Yang, Mengya Liu, Zhumei Wang, Yu Liu, Xiaochen Zhang, Yixuan Li, Le Ma, Yuhang Yang, Weiqi Huang, Wei Liang, Baoxiong Jia, Siyuan Huang
Abstract
Learning generalizable humanoid loco-manipulation remains bottlenecked by costly real-world demonstrations. Simulation provides a scalable alternative, but requires visually faithful, physics-ready scenes and executable whole-body motions. We present Zero-HLM, a real-to-sim-to-real framework that transforms smartphone captures and simulated source demonstrations into scalable training data for humanoid loco-manipulation. Gaussian-guided agentic real-to-sim reconstruction aligns visual and physical scenes; motion-environment co-adaptation uses rollout feedback to correct whole-body references and interaction geometry. We generate 4,800 physically validated demonstrations for zero-shot transfer across three Unitree G1 tasks. Simulation-trained GR00T N1.7 achieves 66.7% mean full-task success, versus 46.7% with real-data training. On generalized configurations, expanded simulation training raises pooled success from 23% for base-trained real-data policies to 69%, without additional real demonstrations. Project page: https://young-bit.github.io/zero-hlm/