Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
2026-08-03 • Robotics
Robotics
AI summaryⓘ
The authors created Ego2Robot, a method to turn videos of humans doing tasks from their own viewpoint into training data for robots. This process includes adjusting human actions to fit robot movements and making realistic robot visuals. They produced a huge dataset covering many robot types and tested how well robot models trained with this data generalize to different situations. Their results show that training with Ego2Robot data helps robots perform better when encountering new or changed environments, even on real robots.
egocentric videosrobot manipulationaction retargetingvisual synthesispretraininggeneralizationvision-language-action modelsrobot morphologiesout-of-distributionrobot deployment
Authors
Ye Wang, Pei Lin, Xiong-Hui Chen, Haoqi Yuan, Zhixuan Liang, Yiyang Huang, Anzhe Chen, Zixing Lei, Jie Zhang, Tao Zhang, Haoyang Li, Tong Zhang, Chenxi Xiao, Ziyuan Jiao, Qin Jin
Abstract
Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present \textbf{Ego2Robot}, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/