Model edits multi-instrument music mixes from MIDI control
Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioning
SoundMachine Learning
Summary
Music often needs small changes without redoing everything. The authors created a model called SpanSynth-Edit that helps change parts of a multi-instrument music recording guided by MIDI notes. It breaks music into simple audio chunks linked to musical notes, letting it resynthesize only the parts you want to edit. Tests show it works well on music with one or many instruments and can control when notes start within short time frames. The authors also point out challenges in judging how well these edits match the original notes based on current methods.
What this means in practice
- •For music producers: Edit and resynthesize only selected parts of multi-instrument recordings directly guided by MIDI changes.
- •For game audio developers: Create responsive multi-instrument audio that updates dynamically based on MIDI-like input for interactive music scenes.
Authors
Sungkyun Chang, Keshav Bhandari, Simon Dixon, Emmanouil Benetos
Abstract
Music creation often involves iterative refinement, changing selected musical details while retaining the rest. To support such refinement, we introduce SpanSynth-Edit, a flow-matching model for MIDI-guided synthesis and editing of multi-instrument audio mixtures using low-frame-rate scalar-quantised latents. MIDI Span encodes instrument-labelled note lifecycles as unordered event sets with continuous-valued attributes and pools each set into one conditioning vector per audio-latent frame. The model uses contextual audio for instrument-specific timbre guidance and supports editing by resynthesising the target region from revised MIDI. Experiments on single- and multi-instrument benchmarks show competitive performance and demonstrate within-frame onset control. We also discuss limitations of transcription-based note-adherence evaluation.