Papers for

virtual reality content creators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

MoVT improves 3D motion creation from text using video data

MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation

Abstract: Text-driven 3D human motion generation models face significant challenges in responding to diverse and unconstrained textual prompts, primarily due to the limited availability of 3D motion training data. To address this, we introduce MoVT, a novel framework that effectively leverages the extensive range of human action videos to enhance text-to-motion generation. At the core of our approach is the cross-modal augmented motion tokenizer, which projects discrete 3D motion tokens into the 2D domain. This projection allows us to enrich the motion codebook with complex, real-world motion patterns derived from videos. The enriched discrete tokens are then mapped back to the 3D domain, resulting in aligned 3D and 2D codebooks with an enhanced capacity to represent intricate motions. These enhanced codebooks are integrated into a generative masked transformer, which predicts masked motion token indices in a modality-agnostic manner. This enables the use of text-index pairs, generated from the 2D codebook and annotated motion videos, to further enhance the generator. Extensive empirical evaluations show that MoVT performs favorably against prior state-of-the-art methods across multiple key metrics.

Mon 14 SeptComputer Vision and Pattern Recognition
The gist
Generating 3D human motions from text is hard because there isn’t enough 3D motion data to learn from. The researchers created MoVT, a new system that learns from lots of regular videos of people moving to better understand and create motions from text descriptions. MoVT converts 3D motions into 2D forms to gather more complex real-world movements from videos and then combines these back into 3D motions. This method helps produce better and more detailed 3D human motions based on text instructions.
Open 2609.14965v1