Long-term text driven human motion modeling improves with new dataset
ELMA: Benchmarking Anaphoric Compositions for Long-Term Text-to-Motion Generation
Computer Vision and Pattern Recognition
Summary
Generating realistic human movements from text instructions is difficult, especially when the instructions span multiple steps that refer back to earlier parts. The authors created a new dataset called ELMA that contains long video clips with detailed labels showing how movements evolve over time. Using ELMA, they improved existing computer models to better handle these linked instructions and produce more realistic and accurate motions. This work highlights where current methods struggle and provides a better starting point for future improvements.
What this means in practice
- •For game developers: Create game characters that perform complex multi-step actions accurately responding to long text instructions including references to earlier movements.$Commercial implications: Enables production of more believable game avatars that follow narrative scripts with continuity, improving player immersion and requiring less manual animation.
- •For animation studios: Generate continuous and context-aware human motions from storyboards or direction text for long sequences without manual keyframing.
Authors
Shun Kato, Dixin Yang, Kotaro Amaya, Yuiga Wada, Kazumi Fukuda, Takashi Shibuya, Yuki Mitsufuji, Mariko Isogawa
Abstract
Text-guided long-term human motion generation is essential for applications such as continuously operating digital avatars. Despite recent progress on short clips, existing models struggle with multi-segment instructions that involve anaphora, where constraints established in earlier segments must be preserved over time. In this paper, we introduce a novel task of long-term motion generation from such anaphoric compositions. To mitigate this, we present ELMA, an episodic long-term motion dataset with dense segment-level annotations that preserve cross-segment dependencies. Built via a fully automated collection and filtering pipeline, ELMA extracts high-quality, continuous 30-second clips from in-the-wild videos, capturing the episodic narratives necessary for long-range state modeling. Leveraging this dataset, we fine-tune state-of-the-art motion generation models to incorporate long-range context. Extensive experiments reveal fundamental shortcomings in current models when handling context-dependent instructions. We demonstrate that ELMA enables significant improvements in both global physical realism and semantic alignment, setting a robust new baseline for long-term motion synthesis.