Improved AI method generates human motion from text with faster and better accuracy
ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
Computer Vision and Pattern Recognition
Summary
Turning written descriptions into realistic human movements can help with games and robots, but it’s hard to get detailed motions from complex text. The authors developed a new method to better match language with specific body movements by structuring the motion data more carefully and integrating the retrieved examples directly where the generator understands them. Their improved system produces more accurate and faster results in tests compared to earlier methods. This makes it easier to create natural movements from words in applications like virtual reality and animation.
What this means in practice
- •For game developers: Generate detailed character animations from text descriptions to speed up content creation in games.$Commercial implications: Enables text-driven animation tools for game studios to produce diverse motions without manual keyframing.
- •For robotics engineers: Convert human instructions into precise joint movements for robots to perform complex tasks more naturally.
Authors
Yiran Wang, Zeyu Zhang, Ling Shao, Hao Tang
Abstract
Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal Attention (SSTA) for topology-aware fusion; and Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.