Video motion tokens improve language guided video generation accuracy
ViMoLa: Discrete Motion Tokens for Language-Guided Video Generation
Computer Vision and Pattern Recognition
Summary
Creating videos from written descriptions often shows the right scenes but struggles to produce the correct movements. The authors introduce ViMoLa, a system that turns language into a sequence of motion tokens, which are then used to guide video creation. Their method improves how well generated videos match the requested motion, by learning from large video datasets. However, it does not perfectly capture all movements yet, and some improvements remain possible.
What this means in practice
- •For interactive media developers: Incorporate motion priors from text to create videos with more accurate movements reflecting user prompts.
- •For special effects artists: Enhance video editing workflows by generating motion sequences matching textual directions for animation or effects.
Authors
Bishoy Galoaa, Sarah Ostadabbas
Abstract
Language-guided video generation often captures scene content while missing the requested motion. We present ViMoLa (Video Motion Language), which predicts a motion prior from language and injects it into a frozen image-to-video backbone. A spatial vector-quantized autoencoder compresses dense trajectories into 64 tokens from a 512-entry codebook; a 28M-parameter transformer predicts the tokens from text; and zero-gated cross-attention adapters condition every frozen Wan~2.2 DiT block through forward hooks. Trained on 462K videos from Something-Something~v2 and Kinetics-400, ViMoLa raises directional accuracy from 0.287 to 0.442 (chance 0.250), with similar measured generative quality. Across-prompt shuffled codes achieve 0.283, supporting the importance of prompt-matched token content. A low-rank adapter reaches 0.428 at 1.45% additional forward-pass FLOPs, and reusing the tokenizer and language model with newly trained CogVideoX adapters raises accuracy from 0.301 to 0.398. Control remains imperfect: oracle codes reach 0.531, compound-motion gains are smaller, and the adapter does not explicitly preserve token-to-patch assignments.