Beatdance creates 3d dances perfectly timed to music beats
BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling
Computer Vision and Pattern Recognition
Summary
Making 3D dance movements that match music beats is tough because the motions must look natural and perfectly synced with the rhythm. The authors designed a new approach called BeatDance that breaks down the dance motion into parts and learns how they change over time more effectively. They also added a way to check that the dance fits the music by trying to rebuild the music from the dance moves, helping the model learn better alignment. Tests show this method creates dance motions that align with music beats better than recent techniques.
What this means in practice
- •For entertainment game developers: Generate realistic 3D dance animations synchronized with music for video games, improving player immersion.
- •For virtual reality creators: Create dance sequences that precisely follow music beats for VR experiences, enhancing realism and engagement.
Authors
Xiaojian Shen, Dahu Shi, Jianrong Zhang, Hai Li, Hongwei Zhao, Dawei Zhang, Yunzhi Zhuge, Zhiliang Wu, Guanghui Yue, Wei Zhou
Abstract
Generating realistic 3D dance from music is a challenging task that requires accurate synchronization with musical rhythms while capturing the spatial complexity of human motion. Although existing methods can generate physically plausible dance motions, they often struggle to achieve precise alignment with music, such as the beat. To address this limitation, we propose a novel diffusion-based framework, BeatDance, with two components: 1) We present a Hierarchical Decoupled Attention (HDA) module, which first disentangles the learning of human pose and temporal dynamics. A hierarchical structure is then employed to capture both short-term and long-term dependencies, thereby enhancing spatial-temporal modeling. 2) We adopt cycle-consistent learning by introducing an auxiliary dance-to-music module. During training, discrepancies between the reconstructed and original music induce a stronger loss signal, effectively encouraging the consistency property between the music and dance motion. Extensive experimental results demonstrate that our proposed approach outperforms recent competitive methods on two benchmark datasets.