Text to music models improve by aligning with symbolic MIDI data
MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment
SoundMachine Learning
Summary
Generating music from text is tricky because machines usually learn directly from raw audio, which is complex. The authors show that teaching the model using simpler symbolic music data, like MIDI, can improve its understanding of melody, rhythm, and harmony. They created a method called MIDI-REPA that combines this symbolic information during training without needing extra data or expensive hardware. Their results suggest that using symbolic music as guidance is an easy and effective way to make text-to-music generation better.
What this means in practice
- •For music software developers: Add symbolic MIDI supervision to improve AI-based music composition tools trained on text prompts.
- •For game audio designers: Generate fitting background music from textual descriptions with improved musical coherence using symbolic alignment.
Authors
Song-Ze Yu, Fang-Duo Tsai, Yi-Hsuan Yang
Abstract
State-of-the-art text-to-music models increasingly rely on scaling model capacity, data, and compute, while still learning musical structure directly from dense audio. Yet melody, rhythm, harmony, and structure can be represented compactly in symbolic form. We introduce MIDI-REPA, a representation-alignment framework that derives MIDI supervision from automatic music transcription (AMT) to align intermediate representations of a flow-matching DiT. Building on the 122M-parameter FluxAudio-S baseline from the 2026 ICME Academic Text-To-Music Challenge, we investigate what, where, and when to align. Aligning a MuScriptor-derived piano roll with an early DiT layer throughout the full flow trajectory performs the best. Without additional training data or techniques, MIDI-REPA remains competitive on the ATTM objective metrics while remaining trainable on a single RTX 3090 in about 31 hours. Our results highlight symbolic supervision as a simple and inexpensive learning signal for improving text-to-music generation. We release our transcription pipeline, JamendoMIDI, training code, and audio samples at https://midi-repa.vaclis.net.