Sonic Palette predicts changing color themes that match music
Sonic Palette: Predicting Dynamic Color Themes from Music
MultimediaSound
Summary
Matching colors with music is tricky because both change over time. The authors developed Sonic Palette, a system that listens to music and predicts a sequence of color themes that evolve alongside the audio. It uses a type of AI model called a Transformer trained on videos where music and color changes are connected. This helps create visuals that naturally follow a song's mood and timing.
What this means in practice
- •For video editors: Generate dynamic color themes that align with music to enhance music video aesthetics.
- •For digital artists: Create evolving visual art pieces that reflect musical atmospheres through time-varying colors.
Authors
Wen-Li Wei, Kenneth Yang, Jen-Chun Lin
Abstract
We present Sonic Palette, a music-driven framework for predicting time-varying color themes from audio. Rather than treating music-to-palette generation as a static problem, our method formulates music--color correspondence as a cross-modal sequence prediction task that models how palette-level visual attributes evolve over time in response to music. Specifically, Sonic Palette uses a music-conditioned autoregressive Transformer to predict frame-wise color themes, each represented by five dominant colors, from input audio features. To support this task, we curate a dataset of paired audio--color sequences derived from 265 full-length official music videos. We further visualize the predicted color themes through a palette-guided recoloring process that preserves image structure while revealing temporally varying color appearance associated with the music. Quantitative results show that Sonic Palette predicts color themes that more closely match the ground truth, while subjective results indicate more natural temporal color transitions and stronger perceived music--color atmosphere alignment, highlighting its potential for music-driven visual design, computational art, and creative visual authoring.