World2Motion generates 3D human motion from images and text prompts
World2Motion: Turning Video World Models into 3D Human Motion Generators
Computer Vision and Pattern Recognition
Summary
Generating realistic 3D human movements that fit specific scenes and texts is hard because current methods struggle when environments vary. The authors build on a video-based world model to directly create 3D motions along with video from just one image and a text description. They overcome challenges like limited paired training data and motion instability by combining synthetic and real data and introducing a new way to reduce jitter. Their method works better than existing 3D motion generators and is faster than previous multi-step approaches.
What this means in practice
- •For game developers: Generate realistic 3D character animations from scene images and text descriptions without needing extensive motion capture data.
- •For robotics teams: Create human motion trajectories aligned with environmental context for robots interacting with people in varied real-world settings.
Authors
Tu Fangyuan, Xiangyue Zhang, Yiyi Cai, Yichen Peng, Kunhang Li, Bo Zheng, Zhixiang Wang, Kaipeng Zhang, Erwin Wu, Haoran Xie, Haiyang Liu
Abstract
We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video--motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video--motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion--text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3$\times$ faster inference.