Music structure analysis improves using audio and lyrics data together
Structural Context from Audio and Lyric Evidence for Music Structure Analysis Using the New SheetSage-MSA Dataset
Sound
Summary
Music structure analysis is about finding the different parts of a song, like verses and choruses, and where they start and end. The authors noticed that existing tools often work only with audio or use the same approach for different sub-tasks, which can limit accuracy. They created a large new dataset with detailed labels and developed a method called SCALE that uses both audio and lyrics to better detect song structure. This method outperforms previous systems on standard tests, showing that combining sound and words helps music understanding.
What this means in practice
- •For music streaming services: Automatically segment songs into meaningful sections to improve music recommendations and playback features.$Commercial implications: Enables enhanced user experience by recognizing song parts, allowing targeted recommendations and smarter playback in streaming platforms.
- •For music software developers: Incorporate audio and lyric-based structure detection to build tools for music editing, analysis, or education with more accurate section labeling.
Authors
Hongbing Li, Yaolong Ju
Abstract
Music structure analysis (MSA) aims to identify functional labels and their boundaries in music. Despite recent progress, three challenges remain. First, fine-grained annotations for MSA are scarce. Second, existing systems often share one encoder between boundary detection and functional labeling despite their different information requirements. Third, most end-to-end systems are audio-only and ignore structural cues from lyrics. To address these issues, we construct SheetSage-MSA, an MSA dataset derived from SheetSage, containing 13,880 cleaned excerpt annotations from 9,010 songs and approximately 82.5 hours of annotated audio. We further propose SCALE (Structural Context from Audio and Lyric Evidence), which uses separate task-specific audio pathways to model local transitions and long-range structural context, and incorporates timestamped lyrics and lyrical recurrence to refine chorus predictions. On the HarmonixSet benchmark, SCALE outperforms the existing state-of-the-art SongFormer by 3.5 and 4.2 percentage points in HR.5F and HR3F, respectively, while improving ACC by 0.9 points. An interactive demo page containing the source code is also available at https://multimodal-music-research-lab.github.io/SCALE/.