What Are You Listening to? Temporal Music Grounding for Audio-to-Text Large Language Models
SoundInformation Retrieval
Summary
The authors study if large audio-language models can accurately point out the exact timing of specific musical notes or patterns in a piece of music. They created MusicGroundingBench, a test set using piano MIDI turned into audio, so they know exactly when each note plays. This test has two parts: one checks if models can find single notes in short clips, and the other checks if they understand short musical phrases. Their experiments show that current models struggle with this task, but training them specifically for it helps a lot. The authors suggest that MusicGroundingBench can help measure how well models truly connect their answers to actual music sounds over time.
audio-language modeltemporal groundingMIDImusic transcriptionsymbolic-to-audio alignmentbenchmarkmusic understandingpiano MIDIaudio groundingstructured grounding
Authors
Kun Fang, Ziyu Wang, Ichiro Fujinaga
Abstract
Large audio-language models can produce fluent and musically plausible responses, yet it often remains unclear whether those responses are grounded in the audio input. We introduce temporal music grounding, a task in which a model returns one or more time spans corresponding to a queried musical note, event, or pattern. To evaluate this capability, we present MusicGroundingBench, a controlled benchmark suite built by rendering algorithmically generated piano MIDI to audio, yielding exact symbolic-to-audio alignment. The suite comprises two subsets: MGBench-3N, which evaluates note-level grounding in clips containing up to three notes, and MGBench-2B, which evaluates structured grounding and short-form music understanding in two-bar excerpts. Experiments show that temporal music grounding remains challenging for current audio-language models, whereas task-specific training yields substantial gains. We further report exploratory evidence on the relationship between grounding supervision and music understanding. These results establish MusicGroundingBench as a controlled testbed for assessing whether audio-language models ground their responses in temporally localized musical evidence.