Controllable music generation enhanced by audio-conditioned cache adapters
DiffSynth-Music: Audio-Conditioned KV-Cache Adapters for Controllable Music Generation
Sound
Summary
Making computer-generated songs sound more like what people want is hard because text instructions don’t capture details like rhythm or style well. The authors created DiffSynth-Music, a system that learns to control music creation by listening to different audio cues like beats, singing, or instruments. They use a special memory technique to mix these audio clues so the music better follows specific rhythms, melodies, or vocal styles. Tests showed it works well for both Mandarin and English songs, improving how accurately the music matches the desired controls and lyrics.
What this means in practice
- •For music producers: Generate music tracks with precise control over beats, vocals, and accompaniment using audio cues for tailored creative outputs.
- •For audio software developers: Integrate advanced audio-conditioned control layers into music creation tools to enhance flexibility and user-guided synthesis.
Authors
Zhongjie Duan, Shengchuan Gao, Hong Zhang, Yingda Chen
Abstract
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.