Open stem audio dataset enables advanced music AI training

SPDMX: Large-Scale Open Stem Audio From Symbolic Music

SoundArtificial IntelligenceMachine LearningMultimedia

Summary

Music AI research struggles because it needs isolated instrument tracks (stems), but music recordings are often copyrighted and not shared publicly. The authors created a large public collection of synthetic stems by converting public-domain music scores into high-quality audio, fully open to everyone. This dataset contains hundreds of thousands of songs with their stems and aligned MIDI information, allowing AI systems to learn tasks like separating instruments, generating music from MIDI, and transcribing music. Their work releases the dataset, metadata, and the tools used so others can build on it.

What this means in practice

  • For music technology developers: Train AI models to separate individual instruments from mixed audio using a large open dataset of synthetic multitrack music.
  • For game sound designers: Generate customizable music stems that adapt dynamically in games by using the openly licensed dataset and rendering tools.

Authors

Phillip Long, Eduardo Escoto, Julian McAuley, Zachary Novack

Abstract

Large public datasets have driven recent progress in machine learning, from text and vision to speech and general audio. Music remains an outlier, as copyright blocks redistribution of training audio, while many modern tasks need individual stems rather than mixtures. Existing multitrack resources trade off realism, scale, and licensing, while synthetic alternatives rely on proprietary VSTs or naive open rendering that yields weak stems. We introduce SPDMX, a listening-guided rendering of public-domain PDMX scores into an openly-licensed stem corpus with aligned scores and an open render pipeline: $253{,}982$ songs, $490{,}159$ mono stems, and $5{,}469$ song-hours at $44.1$ kHz under CC BY 4.0, with paired MIDI and identifiers linking to PDMX metadata. $88{,}403$ songs ($34.8\%$; $2{,}734$ song-hours) provide multitrack supervision. To our knowledge, this is the largest openly-licensed synthetic multitrack subset by song-hours, enabling source separation, stem-conditioned synthesis, MIDI-to-audio, transcription, and joint symbolic-audio modeling. We release the corpus, metadata, and open pipeline for reuse on other symbolic collections. Demos and experimental notes are on the project page.