Text to motion generation improved with smarter retrieval and fusion methods
ReMoMask-2: Latent Retrieval-Augmented Masked Motion Generation
Computer Vision and Pattern Recognition
Summary
Text-to-motion generation helps computers turn written descriptions into human-like movements, useful for games and virtual reality. The authors found that previous systems struggled to connect detailed language with the right motion in a way that respects how the body moves over time and across parts. Their new approach, ReMoMask-2, uses a better way to search and combine motion data that matches the generation process more closely. This results in more accurate and faster creation of motion from text, tested on several standard datasets.
text-to-motion generationretrieval-augmented generationcontrastive learningspatial-temporal topologylatent spacemask transformerFID (Fréchet Inception Distance)HumanML3DKIT-MLSnapMoGen
Authors
Yiran Wang, Zeyu Zhang, Ling Shao, Hao Tang
Abstract
Text-to-motion (T2M) generation maps natural language to human joint movements, aiding gaming, VR, and robotics. Retrieval-Augmented Text-to-Motion (RAG-T2M) improves generation on complex descriptions by conditioning on retrieved motion-text pairs. However, existing RAG-T2M models face two challenges: coarse-grained retrieval and fusion mechanisms overlook the hierarchical, spatial-temporal topology of human motion, and a representation gap exists because retrieved evidence resides in a semantic space separate from the generator's latents. To address the first, we present ReMoMask, a structure-aware RAG framework coupling Hierarchical Bidirectional Momentum (HBM) contrastive learning to align global and part-level features with text; Semantic Spatial-Temporal Attention (SSTA) for topology-aware fusion; and Topology Structured Masking (TSM) to force robust part-level grounding via adaptive masking. To address the second, we introduce ReMoMask-2, which rebuilds the retrieval database directly within the generator's pre-quantization latent space and aligns text queries via a distilled lightweight projector, allowing the generator to directly consume the retrieved motion's semantic content. Extensive experiments on HumanML3D, KIT-ML, and SnapMoGen demonstrate our retriever achieves state-of-the-art accuracy, while ReMoMask-2 attains the lowest FID on KIT-ML and SnapMoGen; notably, its single mask-transformer stage surpasses ReMoMask's full two-stage pipeline and delivers the fastest inference.