MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generation
2026-08-10 • Sound
SoundArtificial IntelligenceMultimedia
AI summaryⓘ
The authors address a problem in text-to-music generation, where current systems don’t clearly show or let users control the structure of the music before it is made. They create MusicLayout, a clear plan that maps out parts of the song like sections, instruments, and patterns in time, acting as a middle step between the text instructions and the music output. Their model first makes this plan, then creates the music based on it, allowing users to see and change the structure beforehand. Tests show that this approach helps organize music better over a longer period and gives more control over how the music is arranged.
text-to-music generationMusicLayoutmusical structureautoregressive modelaudio tokenslayout-conditioned generationmusical arrangementlong-range organizationintermediate representation
Authors
Shuyu Li, Kejun Zhang, Jiahe Lei, Shulei Ji, Zihao Wang, Jiaxing Yu, Wanying Wu, Lei Wang
Abstract
Text-to-music generation has advanced rapidly, but current systems still rely primarily on global text prompts, leaving the structural organization of generated music implicit and difficult to inspect, control, or revise before audio generation. To address this issue, we introduce MusicLayout, an explicit intermediate representation for controlling musical structure in text-to-music generation. MusicLayout describes a musical piece as a time-aligned layout of sections, textures, repetitions, variations, and instrument-level arrangements, serving as an interpretable planning layer between textual intent and the generated music. We integrate MusicLayout into a text-to-music framework built on a unified autoregressive formulation, where the model first generates a MusicLayout representation and subsequently predicts audio tokens conditioned on this representation within a single sequence. The resulting MusicLayout can be inspected and modified prior to audio generation, providing a mechanism for layout-level structural control. We evaluate MusicLayout through layout-conditioned generation, layout manipulation experiments, and matched-data ablations, providing evidence that explicit layout planning can improve long-range structural organization and support layout-level control.