World model predicts human and robot movement for better interaction
Ropedia W0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience
Computer Vision and Pattern RecognitionRobotics
Summary
Predicting how people and robots move and act is hard because they move differently and there isn’t enough combined data. The authors created Ropedia W0, a model trained on lots of human and robot movement videos with detailed annotations. This model understands both walking and hand actions for different bodies and helps machines predict future movements better. It works well on various tests and can even support real-time virtual reality interactions. This could help robots and humans work together more smoothly.
What this means in practice
- •For robotics developers: Use the model to improve robot planning and prediction across different robot types and human interactions with limited retraining.
- •For virtual reality creators: Enable live, minute-long VR experiences that predict realistic user and virtual character movements based on human data.
Authors
Tianqi Liu, Hanfei Zhao, Qifeng Liang, Hui En Pang, Jiarui Xu, Sayang Mu, Penghao Qiu, YingSheng Zhang, Jun Wang, Ziang Cao, Mingze Sun, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
Abstract
World models enable embodied agents to anticipate action consequences, yet jointly modeling locomotion and manipulation across diverse embodiments remains challenging due to heterogeneous action spaces and limited multimodal supervision. We present Ropedia W0, the first egocentric loco-manipulation world foundation model spanning humans and robots with diverse embodiments. We curate Xp-Data, comprising 10k hours of egocentric human data and 3k hours of robot data with unified multimodal annotations and systematic quality control. A unified kinematic representation encodes locomotion through camera poses and manipulation through projected skeletons, with robot skeletons combining embodiment-specific forward kinematics with extensive manual keypoint annotation and correction. We also introduce Xp-Teleop, a long-horizon loco-manipulation benchmark based on high-quality dexterous humanoid teleoperation data. Evaluations demonstrate state-of-the-art performance across five cross-embodiment benchmarks (DreamDojo, $Ψ_0$, $ω$-HOME, Xp-Teleop, and AgiBot) and on held-out human egocentric video, from zero-shot inference to few-shot and full-data post-training. Scaling human pre-training data improves performance on both human and robot evaluations. After distillation, our model supports real-time, minute-long rollouts, enabling live VR interaction. Applications in policy evaluation, model-based planning, and world-action modeling further demonstrate the model's utility for embodied decision-making.