Controllable ai composes music using injected motifs with high accuracy
Getting Motif-ated: Controllable AI Compositions from Injected Motif Prompts
Machine LearningArtificial IntelligenceMultimediaSound
Summary
Music generation AI can create entire pieces from short prompts, but controlling exactly what the music contains is difficult. The authors created MotiGen, a method to steer AI music by adding specific musical motifs as structured prompts. Their approach teaches the model to focus on these motifs, resulting in music that includes the motifs in over 92% of outputs. This method works by fine-tuning existing models with a two-step process, first on short excerpts then full scores.
What this means in practice
- •For music software developers: Add motif control to AI-powered music composition tools for tailored symbolic score generation incorporating user-specified musical phrases.
- •For advertising jingle creators: Generate jingles embedding brand-specific musical motifs on demand to improve brand recognition with AI-assisted composition.$Commercial implications: Enables selling AI composition services embedding custom motifs for branding in advertising music.
Authors
Chao Peter Yang, Cynthia Rudin, Yue Jiang, Simon Mak, Stephen Ni-Hahn
Abstract
Deep learning has transformed symbolic music generation by borrowing the training paradigms of large language models, with systems such as NotaGen now producing complete, stylistically convincing classical scores from a short prompt. These systems could become powerful creative partners, helping musicians generate endless possibilities. However, current systems expose almost no control handles on the music itself. In principle, control handles could be built into a foundation model trained from scratch, but this is rarely practical without massive amounts of quality annotated data and compute resources. We therefore present MotiGen, a recipe for retrofitting pretrained symbolic music models to use new instruction prompts. MotiGen injects a musical motif as a structured prompt line, reinforces it with a scalar attention bias toward the motif tokens, and learns the association with a two-phase curriculum. First, it learns from focused excerpts cropped around motif occurrences in the training data, then full scores including the motifs. Our experiments show that our model composes with the prompted motif in over 92.3\% of generated pieces. Generated pieces using a variety of motifs are included in our sample site: https://motigen-site.github.io/.