Latent world action model improves humanoid robot control
Being-M0.7: A Latent World-Action Model for Humanoid Robots
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Humanoid robots need to move and manipulate objects smoothly, but teaching them is hard because there aren’t many robot demonstrations. The authors made a system called Being-M0.7 that learns from lots of human videos and motion data, even when these datasets don’t match perfectly. This system then adapts what it learned to control robots, helping them predict how the world and their own bodies will move. Their approach improves success in robot tasks involving walking and manipulating objects.
What this means in practice
- •For robotics engineers: Improve humanoid robots’ coordinated walking and object manipulation by using learned visual and motion patterns from human data.
- •For robot software developers: Integrate latent world-action models in robot control systems to enhance prediction of future states and generate whole-body commands.
Authors
Junpeng Yue, Boyuan Li, Yuxuan Wang, Zepeng Wang, Yuhui Fu, Feiyang Xie, Yu Zhang, Jing Zhang, Xianqi Zhang, Weibo Li, Xiaofei Zheng, Yuming Fang, Jiangxing Wang, Zongqing Lu
Abstract
Humanoid loco-manipulation requires coordinated locomotion and manipulation informed by future scene evolution and whole-body motion, yet learning these capabilities is constrained by scarce robot demonstrations. Human video and motion datasets offer scalable supervision, but many contain only video or motion rather than paired video-motion data. Moreover, human motion does not directly specify executable robot actions. We present Being-M0.7, a latent world-action model that transfers visual-motion priors learned from mixed-modality human data to humanoid control through pre-training, robot mid-training, and action post-training. We curate a corpus from more than 10,000 hours of raw human-centric data, integrating video-only, motion-only, and paired video-motion streams to learn complementary visual dynamics and whole-body kinematic structure. Joint prediction of future latent visual states and motion encourages visual representations to encode future kinematics. Robot mid-training adapts this coarse-grained prior to robot viewpoints and body dynamics. During action post-training, an action expert combines visual predictive representations from the frozen, adapted prior with current images and proprioception through gated cross-attention, grounding predictive context in executable whole-body commands. Being-M0.7 achieves the highest aggregate success rate among the compared baselines on SIMPLE and matches the strongest baseline on real-world Unitree G1 loco-manipulation tasks.