Music generative models signals predict human quality ratings
Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals
Sound
Summary
Music-generating AI models can create good songs, but it's unclear if their 'understanding' matches what humans appreciate in music. This paper shows that certain internal signals inside these models, like how surprised they are by what comes next or specific hidden features, strongly relate to how people rate the music. The researchers used these signals from a model called MusicGen to train a small system that can predict how humans would judge a music piece. Among the signals, the features learned by a special autoencoder inside the model best predicted human opinions.
What this means in practice
- •For music software developers: Include model-internal signals to automatically estimate listener quality ratings for newly generated music samples.
- •For audio quality assurance teams: Use lightweight models based on generative model signals to prioritize music outputs that align better with human preferences for curation or production.
Authors
Xiaosha Li, Chun Liu, Ziyu Wang
Abstract
Current music generative models can produce high-quality music, but does this ability imply that they ``understand'' the musical qualities of their outputs, and is that understanding aligned with human evaluation? Previous attempts to use the likelihood of a generative model to evaluate music, an approach commonly used in text, have proven unsuccessful, leading researchers to rely on standalone supervised music evaluation models. In this paper, we answer this question affirmatively: we show that a model's intrinsic signals---derived from its hidden representations and predictions---are strongly correlated with human ratings. In particular, we study MusicGen and consider three types of features: (1) prediction loss, (2) prediction entropy, and (3) concepts extracted from the model using a sparse autoencoder (SAE). Using these features, we train a lightweight prediction model to estimate subjective ratings. We evaluate these features both individually and in combination. We hypothesize that these signals parallel the listening process: the temporal and frequency-domain structure of loss and entropy reflects listeners' expectation and surprise, while gradient directions in SAE latent space predict perceived quality. Experiments on five human-evaluation benchmarks spanning continuous ratings and pairwise preferences confirm this hypothesis, with SAE latents carrying most of the predictive signal.