Timo improves human motion generation from text using multimodal transformers
Timo: $\textbf{T}$aming Mult$\textbf{i}$modal Diffusion Transformer for Human $\textbf{Mo}$tion Generation
Computer Vision and Pattern Recognition
Summary
Generating human movements from text descriptions is a challenge because motions are complex and need to smoothly coordinate different body parts. The authors show that existing methods do not fully understand the connection between text and motion. They introduce Timo, a new system that better models this relationship by combining advanced transformers with special techniques that respect human joint motions and rotations. Their method creates more natural and detailed movements and performs much better than previous approaches on a large, diverse test set.
What this means in practice
- •For game developers: Create realistic character animations from written action descriptions to streamline game content creation.
- •For film animators: Generate detailed and natural human motions from screenplay text for previsualization and animation workflows.
Authors
Zhao Wang, Jiangtao Hu, Jack Yu, Tao Yu
Abstract
Most existing human motion generation (HMG) methods use cross-attention modules to inject text semantics, but ignore the importance of bidirectional modeling between motion and text tokens, which limits text comprehension. A straightforward idea is introducing multimodal diffusion transformers (MMDiT), which have shown effective joint text--visual modeling in vision generation, into HMG. However, we find that articulated motion is temporally coherent but weakly correlated across joints, in which directly applying an MMDiT with flow matching produces poorly coordinated and jerky motion. In this work, we propose Timo, a novel kinematics-aware MMDiT framework tailored for HMG. Timo combines fully shared multimodal attention for bidirectional text--motion modeling with flow matching, geometric and rotational-kinematics supervision that compares actual rotations and their changes over time, and a two-stage curriculum progressing from broad motion learning to detailed caption alignment. Further, we construct a benchmark of $40{,}025$ held-out clips from six public datasets spanning diverse actions, assessing six complementary dimensions under a common evaluator and scoring protocol. Our model substantially outperforms state-of-the-art methods in both quantitative and qualitative evaluations. Remarkably, Timo surpasses Kimodo on five of six dimensions, achieving a $40.8$% relative improvement in the average benchmark score. Project page: https://kyfafyd.wang/projects/timo. Demo page: https://timo.kyfafyd.wang.