Large language models improve music emotion prediction from text and audio
Predicting Music Valence and Arousal via Textual and Multimodal In-Context Learning in Large Language Models
SoundArtificial IntelligenceComputation and Language
Summary
Predicting how music makes people feel is tricky, especially figuring out if the feeling is positive or negative (valence). The authors show that by using large language models and giving them examples to learn from without retraining, they can better guess emotions in songs. They use song lyrics, descriptions of the music, and even raw audio to help improve predictions compared to traditional approaches. Their method works well on a standard test set, especially improving predictions of valence.
What this means in practice
- •For music streaming services: Provide improved emotion-based music recommendations by better predicting continuous valence and arousal from song lyrics and audio at inference time.$Commercial implications: Enables emotionally tailored playlists by integrating emotion prediction without model retraining, enhancing user engagement on streaming platforms.
- •For mobile app developers: Create real-time mood tracking apps that infer listener emotions from lyrics and audio descriptions using large language model in-context learning.
Authors
Chi-En Dai, Chih-Pin Tan, Yi-Hsuan Yang
Abstract
While arousal can be reliably predicted from musical audio, emotional valence remains difficult to model due to its subjective and context-dependent nature. In this work, we investigate the potential of in-context learning (ICL) with large language models to advance continuous valence and arousal prediction. Rather than predicting emotion scores directly, our framework leverages contextual demonstrations from an auxiliary dataset to refine the initial predictions produced by a small, task-specific base model. We systematically evaluate a text-based ICL variant using lyrics and generated audio captions, alongside a multimodal variant that incorporates raw audio signals. Using the entire MER60 dataset strictly as a held-out test set, our experiments demonstrate substantial performance gains, particularly for valence. Text-based ICL improves the valence $R^2$ from 0.110 to 0.427 with 80 demonstration shots, while the multimodal variant achieves 0.346 with only 10 shots. These findings suggest that training-free ICL offers an effective inference-time strategy for current language models to predict continuous music emotion.