P-MUSE: Prompt-MIDI-Optional Model for Unified Instrumental Music Synthesis and Editing
2026-08-03 • Sound
Sound
AI summaryⓘ
The authors present P-MUSE, a system that turns MIDI files into music by combining the melody and rhythm from MIDI with the sound style (timbre) taken from a short audio clip. Their approach works both when you have matching MIDI and audio examples and when you only have audio, making it flexible. They also made it possible to both create new music and edit parts of existing audio using the same method. To guide the system better, they introduced some new techniques based on theory and experiments. Additionally, they created a new benchmark to test such systems using four instruments: piano, guitar, bass, and drums.
MIDIinstrument timbreconditional generationin-context learningcurriculum learningfill-in-the-middleclassifier-free guidancephase-aware schedulingmusic generationmusic editing
Authors
Chong Jing, Junan Zhang, Jing Yang, Yulun Wu, Fan Fan, Zhizheng Wu
Abstract
MIDI-to-Music system renders the melody and rhythm of a target MIDI sequence into musical segment while cloning instrument timbre from a prompt recording. Existing systems typically adopt one of two distinct paradigms: conditional generation with prompt audio alone, which remains applicable when aligned prompt MIDI is unavailable, and In-Context Learning with paired prompt audio and MIDI, which exploits cross-modal alignment for stronger control on MIDI following and timbre similarity. We introduce P-MUSE, an instrumental MIDI-to-Music framework that unifies both paradigms via a multi-stage Curriculum-Learning supporting prompt-MIDI-optional inputs. P-MUSE further unifies music generation and local editing through a shared fill-in-the-middle formulation. Grounded in theoretical analysis and empirical study, we propose a phase-aware classifier-free guidance scheduling principle for Transcription-to-Audio systems, alongside a Tail-Drop strategy. Finally, to advance research in this field, we establish the first comprehensive benchmark, covering various prompt modes, generation/editing tasks, and four representative instruments: piano, guitar, bass, and drums. Demos are available at https://p-muse.github.io/.