Large audio language models understate timing details in music prompts
Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models
SoundComputation and Language
Summary
Large audio-language models can understand what instruments are playing but have trouble pinpointing the exact moment something changes in the music. The study found this difficulty depends on how the models are asked and how much music they hear beforehand. Some queries reveal the model knows about the music moment even if it can't say exactly when it happens. The authors show that current prompting methods miss how much temporal understanding the models actually have.
What this means in practice
- •For music software developers: Improve music tagging tools by better probing models to detect musical moment changes accurately within audio segments.
- •For audio content indexing teams: Build enhanced search systems that verify musical events with more temporal context to improve indexing quality for audio archives.
Authors
Eunsik Shin
Abstract
Large audio-language models (LALMs) understand music, yet struggle to locate a described moment in time. We ask whether this reflects missing temporal information or a bottleneck in how the time span is extracted. Using a paired-query probe built from Slakh2100 stem activations, we attach a state query (e.g., ``only piano and bass are playing'') and a change query (e.g., ``the guitar drops out, leaving piano and bass'') to the same span and vary how much preceding audio the model is shown. Across frozen LALMs, three ways of extracting the span ask different questions: full-track prompting asking whether the model can say when; windowed yes/no scoring asking whether it verifies the moment when handed the segment, and whether it needs what came before; attention asking whether it can find the moment on its own. State queries score above baseline without context, while change queries improve with preceding context in most short-segment models; 30s padded encoders show no comparable recovery. Prompting and attention expose almost none of this. Prompting failures thus understate what LALMs can verify, though not what they can locate, by an amount set by query type and encoder.