Dance to music generation improves with hierarchical expert supervision
CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation
Artificial IntelligenceSound
Summary
Generating music that matches a dance video is tricky because dance gives limited clues, while music includes many complex features. The authors developed a new method that uses a music expert model to guide the music generation process at multiple levels, helping bridge the gap between dance and music. They also teach the system step-by-step and align its understanding of music features carefully to improve quality and rhythm matching. Their experiments show better music that fits the dance style and timing well.
What this means in practice
- •For music software developers: Create tools that automatically generate music tailored to specific dance videos with improved rhythm and style alignment.
- •For game audio designers: Produce adaptive in-game music that better matches character movements and dance animations through enhanced generation techniques.
Authors
Jinting Wang, Chenxing Li, Dong Yu, Li Liu
Abstract
Dance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics. Existing methods typically rely on these sparse cues and supervise only the final audio output, resulting in poorly learned music representations and generated music with limited musicality and structural coherence. To address these issues, we propose Curriculum-guided Multi-scale representation Alignment with scale-aware Optimal Transport (CMA-OT), a novel paradigm that leverages an external music expert to provide hierarchical supervision for the generator's latent features, bridging the semantic gap and enhancing representation learning. To effectively incorporate hierarchical supervision, we introduce a curriculum-guided multi-scale learning strategy that progressively transfers musical knowledge from the expert to the music generator, enabling stable and effective representation learning. Moreover, to accommodate the semantic and structural variations across different expert scales and achieve fine-grained alignment under temporal mismatch, we propose a scale-aware optimal transport alignment mechanism, which models soft correspondences between hierarchical expert representations and the generator's latent features. Extensive experiments on two datasets demonstrate that CMA-OT achieves state-of-the-art performance in rhythmic synchronization, perceptual quality, and overall music generation.