Image models generate 3D human motion across different skeletons
Image Diffusion Models Are Good 3D Human Motion Generators
Computer Vision and Pattern Recognition
Summary
Generating 3D human motion from text usually requires models specialized for specific skeleton layouts, limiting their flexibility. The authors show that image-based diffusion models can represent human motion as images encoding joint movements over time, working with various skeleton structures without needing to adjust layouts. Their method combines global and local motion information and can be trained across multiple datasets, achieving state-of-the-art results. This approach enables a single model to generate accurate motion for different types of skeletal data.
What this means in practice
- •For animation studios: Generate diverse 3D character motions from text prompts without remapping skeleton layouts for each character rig.$Commercial implications: Enables creation of customizable animation data for multiple character models, improving production pipelines and reducing manual adjustments.
- •For game developers: Produce realistic human motions from textual descriptions that work across various skeletal rigs to expedite character animation.
Authors
Siyuan Yang, Chong Wang, Hao Cheng, Jun Liu, Bihan Wen, Alex Kot, Hedvig Kjellstrom
Abstract
Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogeneous skeletal layouts while preserving motion-specific kinematic structure. To this end, we organize root-centered joint trajectories into topology-informed joint--time images and model global root dynamics through a complementary latent stream. A motion autoencoder conditions local-motion decoding on the root latent, while a text-conditioned Scalable Interpolant Transformer (SiT) jointly generates both latent streams, with supervision on decoded 3D joint positions further constraining generation in motion space. In addition to dataset-specific models, we train a single motion autoencoder on HumanML3D and KIT-ML, followed by a single text-conditioned SiT shared across both datasets, preserving their respective skeletal definitions without skeleton retargeting to a common joint layout. Experiments on HumanML3D and KIT-ML show that our dataset-specific models achieve state-of-the-art performance, while shared training yields further gains on both benchmarks, demonstrating the benefits of shared modeling across heterogeneous skeletal layouts.