Music identity masking benchmark measures tempo and structure detection
MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models
Sound
Summary
It can be hard to test how well AI models understand music because sometimes answers come from hidden information like song titles, not the sound itself. The authors created a set of 500 music clips with all identifying details removed, focusing only on the sound and annotated features like tempo, key, and rhythm. This benchmark lets researchers check if AI really listens to and understands music details rather than guessing from outside clues. They also show baseline results from simple methods and discuss challenges in making such tests fair and reliable.
What this means in practice
- •For music software developers: Use the benchmark to evaluate how well AI systems identify tempo and structural features from raw audio without relying on metadata.
- •For audio signal processing teams: Test new audio analysis algorithms on a controlled dataset that prevents exploiting metadata for music understanding tasks.
Tested on one dataset.
Authors
Kun Liu, Lyra Tao
Abstract
Evaluating musical perception in multimodal language models requires separating audio evidence from answers recoverable through metadata or external tools. We present the Music Understanding Benchmark (MUB), a locally assembled collection of 500 identity-masked recordings with existing annotations for tempo, key, time signature, and structural intervals. The collection covers all 24 major/minor key combinations, tempos of 47--180 beats per minute, and 4,676 annotated intervals. Original titles, artists, and platform identifiers are excluded from evaluation inputs. Relative to recent music question-answering and temporal-grounding work, MUB emphasizes scalar tempo prediction and full-recording interval outputs, together with a documented metadata-masking protocol. Saved baseline predictions yield 53.6\% tempo accuracy for a simple DSP estimator and 98.4\% time-signature accuracy for a constant 4/4 predictor. We identify important limitations, including an asymmetric legacy key metric, unverified independent annotation agreement, extreme metre imbalance, and unresolved recording-level rights clearance. No contemporary multimodal language model comparison has yet been completed. MUB is therefore an evaluation resource and protocol under validation, rather than a completed demonstration of model capabilities.