SheetSage2 improves music score transcription using synthetic training
SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
Sound
Summary
Transcribing music into readable sheet music is hard because it requires understanding rhythm, melody, harmony, and how music fits together over time. The authors show that using computer-generated music with known notes helps train a model to recognize music better. Their approach, called SheetSage2, combines this synthetic data with special decoding methods to make sure the transcriptions make musical sense from start to finish. As a result, their model produces more accurate and coherent music transcriptions than previous systems on many tests.
What this means in practice
- •For music software developers: Create music transcription tools that convert audio into readable sheet music with improved accuracy and coherence.$Commercial implications: Enables building better commercial music transcription apps that attract musicians seeking reliable score generation.
- •For digital audio workstation creators: Integrate advanced transcription models into DAWs to help users automatically generate lead sheets from recordings.
Authors
Junyan Jiang, Ruibin Yuan, Jiahao Pan, Wei Xue, Yike Guo, Gus Xia, Yann LeCun
Abstract
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.