Large audio models improve music reasoning across song stems and sections
STEMMA: Song-to-Stem Multi-Audio Reasoning for Large Audio Language Models
Sound
Summary
Understanding music often means comparing different parts of songs and how they relate, like which pieces belong together or come from the same track. The authors created STEMMA, a new way to test and train audio-language models so they can better recognize these relationships using song production information. They use a special method to pick examples based on known connections rather than guessing from descriptions. Training models this way helps them reason about multiple audio clips working together while still understanding single pieces well.
What this means in practice
- •For music streaming platforms: Improve song recommendation and playlist creation by recognizing relationships among song sections and stems from the same track.$Commercial implications: Enables enhanced content organization and personalized recommendations by leveraging multi-audio reasoning in large audio-language models.
- •For audio engineering teams: Automate identification of song stems and their relations to mixtures for easier music production analysis and remixing.
Authors
Hoyeol Sohn, Wonil Kim, Keunhyoung Kim, Sangeun Kum, Taehyoung Kim, Dongjoo Moon, Theerasak Charoenchob, Teeratep Weerapang, Jongpil Lee, Juhan Nam
Abstract
Music understanding often requires comparing excerpts and reasoning about relationships among songs, sections, and stems. However, existing large audio-language models (LALMs) and music question-answering datasets typically operate on single recordings or compare independently sampled tracks with no known production relationship. We introduce STEMMA, a multi-audio music question-answering framework built around production provenance: whether excerpts originate from the same track or section, and which stems belong to which mixtures. Because such relations are sparse under conventional audio-first sampling, STEMMA adopts a relation-first construction strategy: it first specifies a target relation and then queries the catalog for excerpts that satisfy it and hard negatives that do not. Labels are determined directly from catalog provenance rather than generated by a language model from metadata. We build STEMMA-Bench for evaluation and a track-disjoint training set, STEMMA-Instruct. Fine-tuning two LALMs on STEMMA-Instruct improves multi-audio reasoning, with the largest gains on structural relations directly determined by the catalog, while preserving single-audio music understanding.