Music text models fail to link instruments and descriptions accurately
Don't CLAP: Are Music-Text Models Bag-of-Words?
SoundComputation and Language
Summary
This paper investigates whether popular music-text AI models truly understand the connection between words describing music and the actual sounds. The authors tested if these models can tell when one musical detail, like an instrument's sound, is swapped with another in the song's description. They found that most models treat the text more like a bag of words without understanding how specific words connect to particular sounds. This means the commonly used metric for judging these models doesn’t reliably reflect how well the music matches the description.
What this means in practice
- •For music app developers: Improve music recommendation systems by recognizing that current text-audio similarity scores may not capture subtle musical details.
- •For audio captioning tool creators: Design better evaluation metrics that test whether models understand detailed musical attributes rather than just word overlap.
Authors
Yuan-Chiao Cheng, Alexander Lerch
Abstract
Text-to-music systems are assessed on audio quality and on how faithfully the music follows its prompt, and the CLAP score, the cosine similarity between a music-text model's audio and text embeddings, is the standard objective metric of faithfulness. We ask how accurately that score reflects the text: when an attribute is linked to an instrument (e.g., distorted guitar), does the text embedding capture that binding? To find out, we introduce an attribute swap perturbation: the caption of a real recording is edited by exchanging exactly one property, timbre, lead versus accompaniment, or order of first appearance, between two instruments. We then test four contrastive music-text models and one large audio-language model on whether the audio scores higher against the original caption than against the perturbed one. No contrastive model distinguishes the two captions reliably. The audio-language model does better, but further experiments show that its advantage rests largely on audio-agnostic language priors. Our results thus provide compelling evidence that the CLAP score and related metrics do not capture fine-grained musical meaning or attribute bindings; their representation is closer to a bag-of-words that leaves them insensitive to meaning-changing perturbations of the caption.