Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address problems in text-driven human motion synthesis by redesigning how motions are represented and generated. They create a model, DeMoDiff, that encodes the movement of each body joint separately instead of all at once, allowing better control of individual parts. Their new method uses a special diffusion process to generate motions that are more accurate and easier to edit in time and space. Tests on standard datasets show their approach improves motion quality and controllability compared to earlier methods.
human motion synthesisvector quantization (VQ)variational autoencoder (VAE)autoregressive diffusionspatiotemporal representationmotion generationattention mechanismlatent spacepart-level controllabilityHumanML3D dataset
Authors
Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao, Weili Zeng, Yichao Yan
Abstract
Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/