Deep learning separates individual notes from complex music recordings
On a Separate Note: Robust Score-Informed Note Separation with a Two-Stream TFC-TDF U-Net and Adaptive Set Ownership
Sound
Summary
Separating individual musical notes from songs where many instruments play together is very difficult. The authors created a deep learning tool called NoteSep that uses the musical score to help isolate each note's sound. Their method splits notes into harmonic and percussive parts and cleverly decides which notes own which sounds to avoid overlaps. They trained and tested their approach on a large, carefully prepared dataset and significantly improved note separation compared to previous methods.
What this means in practice
- •For music producers: Isolate and manipulate individual notes from multi-instrument recordings for remixing or restoration.$Commercial implications: Enables new tools for audio editing companies to offer precise note-level track separation services.
- •For music transcription software developers: Improve automatic music transcription accuracy by extracting cleaner note-level audio from complex mixes.
Authors
Benjamin Shiue-Hal Chou, Purvish Jajal, Nicholas John Eliopoulos, James C. Davis, George K. Thiruvathukal, Kristen Yeon-Ji Yun, Hao-Wen Dong, Yung-Hsiang Lu
Abstract
Score-informed note separation seeks to extract the performed waveform of all individual notes, often from a polyphonic recording. Existing deep learning systems generally only target instrument-level stems. We present, to our knowledge, the first deep learning approach to score-informed note separation, NoteSep. NoteSep extracts the queried notes by applying an extraction stage model, NoteGrab, once per note. Conditioned on pitch, onset, and offset, NoteGrab separates harmonic and percussive components in two U-Nets linked by bidirectional cross-attention; selective harmonic gating suppresses lower-octave interference while preserving percussive attacks. Finally, a joint separation stage applies Adaptive Set Ownership (ASO) to compare concurrent NoteGrab estimates and reallocate mixture energy. We curate SCNS-Train (25,729 mixtures and 743,920 targets) for training and SCNS-Eval (16 instruments, disjoint scores and libraries) for evaluation. On SCNS-Eval, NoteSep reaches a median SI-SDR of 7.39~dB, compared with 2.49~dB for our strongest baseline. See the demo page at https://benschou.com/notesep.