Songcraft model generates and edits songs with fine detail
SongCraft: Unified Song Generation and Editing with Reconstructive Learning
Sound
Summary
Creating songs and changing them have usually been two separate tasks, but this paper shows how the same model can do both. The authors trained their model to rebuild songs using different sets of instructions, like text, lyrics, or detailed musical parts. This lets it both compose new songs and change specific parts like lyrics or melody while keeping the rest intact. They improved the model’s understanding of words and rhythm to make songs that sound clearer and more natural. Their tests show it makes fewer word mistakes than other song generators and can handle many kinds of song edits with one system.
What this means in practice
- •For music producers: Edit specific song elements like lyrics, melody, or singer identity within generated audio using a single integrated model.
- •For audio software developers: Build tools that generate songs from text and lyrics with improved pronunciation and musicality using reconstructive pretraining.
Authors
Haohe Liu, Varun Nagaraja, Gael Le Lan, Xinhao Mei, Zhaoheng Ni, Vikas Chandra, Abdelrahman Mohamed, Yangyang Shi
Abstract
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.