WorksComputer Science
Sonic Palette: Predicting Dynamic Color Themes from Music
Abstract
We present Sonic Palette, a music-driven framework for predicting time-varying color themes from audio. Rather than treating music-to-palette generation as a static problem, our method formulates music–color correspondence as a cross-modal sequence prediction task that models how palette-level visual attributes evolve over time in response to music. Specifically, Sonic Palette uses a music-conditioned autoregressive Transformer to predict frame-wise color themes, each represented by five dominant colors, from input audio features. To support this task, we curate a dataset of paired audio–color sequences derived from 265 full-length official music videos. We further visualize the predicted color themes through a palette-guided recoloring process that preserves image structure while revealing temporally varying color appearance associated with the music. Quantitative results show that Sonic Palette predicts color themes that more closely match the ground truth, while subjective results indicate more natural temporal color transitions and stronger perceived music–color atmosphere alignment, highlighting its potential for music-driven visual design, computational art, and creative visual authoring.
Introduction
Music is inherently temporal, yet its perceptual impact extends beyond the auditory domain. Musical content can evoke vivid visual impressions of color, brightness, and atmosphere, and such cross-modal associations have long influenced artistic practice and perceptual research. For example, gentle passages may suggest muted or warm palettes, whereas energetic passages may evoke more vivid and rapidly changing colors. Prior perceptual studies have provided evidence that music and color are systematically related through affective and acoustic characteristics [1, 2, 3, 4]. However, recent computational work on music–color modeling has focused on predicting a static palette representation from a music input [5]. Such a formulation does not capture how color themes evolve as the musical content changes over time.
This limitation is particularly relevant for applications in which music and visual appearance unfold jointly over time. A single palette may characterize the overall color impression of a piece, but cannot capture local variations as the music evolves. Modeling such dynamic music–color correspondence therefore involves not only learning cross-modal associations, but also capturing temporal dependencies across successive color themes. To address this problem, we propose Sonic Palette, a music-driven framework for predicting time-varying color themes from audio.
Rather than targeting detailed or photorealistic image colorization, Sonic Palette focuses on palette-level visual attributes that compactly characterize color appearance. We formulate music–color modeling as a cross-modal sequence prediction problem: given a temporally ordered sequence of audio features, the model predicts a corresponding sequence of frame-wise color themes, each represented by five dominant colors. The predicted sequence can subsequently be used in a palette-guided recoloring process to visualize how music-conditioned color appearance evolves over time (see Figure 1). Unlike prior music-to-palette generation work that produces a single static palette representation for a music input [5], our formulation explicitly models the temporal evolution of color themes throughout the music.

Figure 1. Overview of Sonic Palette. Given an input audio segment, the framework predicts a time-varying sequence of color themes to drive a palette-guided recoloring process.
To support this task, we curate a dataset of 265 full-length official music videos and derive temporally aligned audio–color sequences from them. Each video frame is represented by a color theme consisting of five dominant colors [6], providing a structured representation of its palette-level appearance. Audio representations are extracted using the pre-trained MERT model [7]. We then train a music-conditioned model based on an autoregressive Transformer [8, 9, 10, 11] to predict the frame-wise color-theme sequence. By conditioning each prediction on both the temporally contextualized audio features and preceding color themes, the model captures dependencies across successive color themes and their evolution in response to music. This autoregressive formulation encourages coherent transitions rather than treating each frame-wise prediction independently.
Our contributions are threefold. First, we introduce a dataset of paired audio–color sequences derived from 265 full-length official music videos, providing a resource for studying music-driven dynamic color theme prediction. Second, we formulate music-to-palette generation as a cross-modal sequence prediction problem, extending static palette generation toward temporally evolving frame-wise color themes. Third, we quantitatively evaluate the predicted color-theme sequences and conduct subjective evaluations of their palette-guided recoloring results, showing closer agreement with the ground truth, more natural temporal color transitions, and stronger perceived music–color atmosphere alignment. Our results highlight the potential of dynamic music–color modeling for music-driven visual design, computational art, and creative visual authoring.
Related Work
Music–Color Correspondence
Palmer et al. [1] found systematic music–color associations that were strongly mediated by perceived emotion; for example, faster major-mode music tends to be associated with lighter, more saturated, and yellower colors, whereas slower minor-mode music tends to be associated with darker, less saturated, and bluer colors. Lindborg and Friberg [2] further examined this relationship using an interface for navigating the CIE Lab color space and showed that models incorporating perceived emotion explained 60–75% of the variation in color responses, providing further evidence for emotion-mediated music–color associations. Whiteford et al. [3] subsequently found systematic music–color associations across a broader range of musical styles, further supporting the role of emotional mediation. More recently, Reymore and Lindsey [4] examined musical timbre–color correspondences, showing that semantic timbre ratings predicted the lightness and saturation of matched colors. Collectively, these studies provide a perceptual foundation for computational modeling of music–color correspondence.
Music-to-Palette Generation
Building on perceptual evidence of music–color correspondence, recent computational work has explored automatic color generation from music. Music2Palette [5] formulates music-to-palette generation as a cross-modal representation learning problem and constructs emotion-aligned music–palette pairs for direct palette generation from music. Its framework jointly promotes emotion alignment, color diversity, and palette coherence, demonstrating the feasibility of learning direct mappings between music and color palettes. However, Music2Palette produces a single static palette representation for a music input rather than explicitly modeling the temporal evolution of color themes throughout a full-length piece. In contrast, our work formulates music–color correspondence as a cross-modal sequence prediction problem, predicting frame-wise color themes that evolve over time in response to music.
Approach
Given paired training data , where denotes a sequence of MERT audio features and denotes the corresponding ground-truth frame-wise color themes, our goal is to model the temporal correspondence between musical content and palette-level visual appearance. We train a music-conditioned autoregressive Transformer to predict a sequence of frame-wise color themes from audio features over time, as illustrated in Figure 1. Each audio feature encodes the musical content associated with frame , and the input sequence is organized as , where and denote the sequence length and feature dimension, respectively. By retaining the temporal ordering of both modalities, the formulation enables the model to learn how successive color themes vary with the corresponding audio context. During training, the audio and its corresponding color-theme sequence are divided into non-overlapping 2-second segments, with each pair corresponding to one such segment. At inference time, the input audio is processed in the same segment-wise manner.
To obtain a compact and structured output space, we represent each frame-wise color theme using five dominant colors [6, 12]. The dominant colors are selected from a quantized RGB histogram in descending order of color frequency, while Euclidean distance-based filtering is applied to avoid selecting near-duplicate colors (see Figure 2). This representation summarizes the major color composition of a frame while avoiding the complexity of directly predicting dense pixel-level appearance. Each dominant color is represented by three RGB channels, with each channel quantized into 16 discrete bins. The quantization therefore converts color prediction from continuous RGB regression into a set of discrete classification problems, yielding a compact palette-level output space for dynamic color theme prediction. Concretely, each frame-wise color theme is represented as a matrix of discrete bin labels, or equivalently as a one-hot tensor.

Figure 2. Examples of frame-wise color themes from Billie Eilish’s “What Was I Made For?” (from Barbie).
Under this formulation, the model predicts the conditional distribution of as , where denotes the preceding color themes and denotes the sequence of temporally contextualized audio features obtained by applying a Transformer encoder [13] to the input sequence of MERT audio features. The training objective aggregates cross-entropy losses over the discrete bin labels of all RGB channels across all dominant colors. At inference time, the highest-probability bin is selected for each RGB channel of each dominant color to form the predicted color theme . Previously predicted color themes are autoregressively fed back to predict subsequent themes (see Figure 1). Repeating this autoregressive process over time yields a predicted sequence of frame-wise color themes of length .
To provide a perceptual visualization of the predicted dynamic color themes, we further apply a palette-guided recoloring process, as shown in Figure 1. Given an input image and a predicted color theme , let denote the quantized RGB color obtained by mapping the predicted bin labels of the -th dominant color back to RGB values. Each pixel in the input image is then assigned to its nearest predicted dominant color according to Euclidean distance in RGB space:
The pixel is subsequently recolored using . Applying this mapping to all pixels preserves the spatial layout and underlying image structure while replacing the original color appearance according to the predicted color theme. Repeating this recoloring process over successive predicted themes provides a direct visualization of how the music-conditioned color appearance changes over time.
Experiments
Dataset and Evaluation Metrics
We curate a dataset of 265 full-length official music videos, comprising approximately 18.5 hours of audiovisual content and 1.7M video frames, and derive temporally aligned audio–color sequences for studying music-driven dynamic color theme prediction. Among these videos, 65 are drawn from the DEAP dataset [14], while the remaining 200 are collected from online sources to broaden the musical and visual coverage of the dataset. To evaluate the model under diverse emotional conditions, we construct the validation and test sets from the DEAP subset, which provides valence–arousal annotations for each video. In all experiments, we use 225 videos for training, 20 for validation, and 20 for testing. Both validation and test splits are balanced across four valence–arousal quadrants, with five videos selected from each quadrant, enabling evaluation across a range of affective conditions. Upon acceptance, we will release source links for the official music videos, along with frame-wise color-theme annotations, to support reproducibility and future research.
We evaluate color-theme prediction quality using Average Color Bin Difference (ACBD). Since the five dominant colors in each color theme are ordered by color frequency, we compare the predicted and ground-truth dominant colors at the same rank. Let and denote the predicted and ground-truth RGB bin labels of the -th dominant color at frame , respectively. We define the color bin difference for a single dominant color as
We then average over all five dominant colors and all frames within a segment:
Lower ACBD indicates smaller discrepancies between the predicted and ground-truth color themes. We report the average ACBD over all test segments.
We also measure temporal variation using Average Temporal Bin Difference (ATBD):
Lower ATBD indicates closer agreement between the predicted and ground-truth temporal variations. We report the average ATBD over all test segments.
Quantitative and Subjective Results
Since existing music-to-palette work is not directly applicable to frame-wise dynamic color-theme prediction, we quantitatively compare our autoregressive Transformer against two task-specific baselines: a non-autoregressive Transformer that predicts each frame-wise color theme using temporally contextualized audio features without conditioning on previously predicted outputs, and a static Multilayer Perceptron (MLP) that predicts a single color theme for an entire song without modeling temporal evolution. All models are optimized using Adam [15] with dropout [16] applied to mitigate overfitting. The static MLP is trained for 100 epochs with a learning rate of . The non-autoregressive Transformer and our autoregressive Transformer are both trained for 50 epochs, with learning rates of and , respectively. As shown in Table 1, our method outperforms the non-autoregressive Transformer baseline, achieving lower ACBD and ATBD. These results suggest that conditioning each prediction on preceding color themes improves frame-wise prediction accuracy and better captures temporal variation relative to the ground truth. We also observe clear improvements over the static MLP baseline, consistent with the limitation of predicting a single color theme for an entire song, which cannot represent frame-wise temporal variation. For fair ACBD and ATBD computation, the color theme predicted by the static MLP is repeated across all frames of the song for evaluation.
| Methods | Average ACBD ↓ | Average ATBD ↓ |
| Static MLP | 4.6757 | 0.9959 |
| Non-autoregressive Transformer | 3.8853 | 1.0068 |
| Ours | 3.5161 | 0.8671 |
Table 1. Quantitative Evaluation of Music-Driven Dynamic Color Theme Prediction
Subjective evaluation using 5-point opinion scores across five evaluation sets, each consisting of three recolored videos paired with the same music, further indicates that our method receives higher ratings than both baselines for Temporal Color Transition Naturalness and Music–Color Atmosphere Match (see Figure 3). This suggests that our recolored results exhibit more natural temporal color transitions with less perceived flicker and stronger perceived music–color atmosphere alignment. In comparison, the non-autoregressive Transformer tends to produce abrupt frame-to-frame changes in predicted color themes, which can lead to noticeable temporal flicker in the recolored video. The static MLP, by contrast, uses a temporally constant predicted color theme throughout the song and therefore does not explicitly model temporal color-theme variation. Although its recolored appearance may still vary across frames because the recoloring process depends jointly on the input image content and the predicted color theme, such variation arises from changes in the input imagery under a fixed predicted color theme, rather than from temporal adaptation of the color theme itself. Consequently, the static MLP cannot directly adapt its color-theme predictions to temporal changes in the music, limiting its ability to model time-varying music–color atmosphere alignment. These differences are also illustrated in the accompanying demo video1. Overall, the quantitative and subjective results consistently support the effectiveness of our autoregressive formulation for dynamic music–color modeling.

Figure 3. Subjective 5-point opinion scores from 26 raters.
CONCLUSION
We presented Sonic Palette, a music-driven framework that formulates music–color correspondence as a cross-modal sequence prediction problem for predicting time-varying color themes from audio. Using a music-conditioned autoregressive Transformer, our method models the temporal evolution of successive frame-wise color themes in response to music. We curate a dataset of 265 full-length official music videos to support this task. Experimental results show that Sonic Palette achieves more accurate color-theme predictions and more closely matches the ground-truth temporal variation, while subjective evaluation indicates more natural temporal color transitions and stronger perceived music–color atmosphere alignment. These results highlight the potential of dynamic music–color modeling for music-driven visual design, computational art, and creative visual authoring.
Acknowledgment
This work was supported by NSTC Grant 113-2221-E-001-017-MY3.
Compliance with Ethical Standards
This study involved an anonymous and voluntary subjective evaluation of recolored videos, and no personally identifiable information was collected.
References
References
- [1]Stephen E. Palmer, Karen B. Schloss, Zoe Xu, and Lilia R. Prado-León, “Music–color associations are mediated by emotion,” Proceedings of the National Academy of Sciences, vol. 110, no. 22, pp. 8836–8841, 2013.DOI
- [2]PerMagnus Lindborg and Anders K. Friberg, “Colour association with music is mediated by emotion: Evidence from an experiment using a CIE Lab interface and interviews,” PLoS ONE, vol. 10, no. 12, 2015.DOI
- [3]Kelly J. Whiteford, Karen B. Schloss, Nathaniel E. Helwig, and Stephen E. Palmer, “Color, music, and emotion: Bach to the blues,” i-Perception, vol. 9, no. 6, 2018.
- [4]Lindsey Reymore and Delwin T. Lindsey, “Color and tone color: audiovisual crossmodal correspondences with musical instrument timbre,” Frontiers in Psychology, vol. 15, no. 1520131, 2025.DOI
- [5]Jiayun Hu, Yueyi He, Tianyi Liang, Changbo Wang, and Chenhui Li, “Music2Palette: Emotion-aligned color palette generation via cross-modal representation learning,” in Proceedings of the 33rd ACM International Conference on Multimedia (MM), 2025.arxiv.org/abs/2507.04758
- [6]Shigenobu Kobayashi, Color Image Scale, Kodansha International, 1991.
- [7]Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu, “MERT: Acoustic music understanding model with large-scale self-supervised training,” in International Conference on Learning Representations (ICLR), 2024.
- [8]Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever, “Generative pretraining from pixels,” in Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
- [9]Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen, “T2M-GPT: Generating human motion from textual descriptions with discrete representations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2301.06052
- [10]Wen-Li Wei, Yu-Hua Hu, and Jen-Chun Lin, “CamSketch: Authoring camera trajectories for 3D virtual concerts via example-derived codes and user sketching,” in Proceedings of the SIGGRAPH Asia 2025 Posters, 2025.DOI
- [11]Jian-Kai Zhu, Wen-Li Wei, and Jen-Chun Lin, “Tai-Chi: Text-to-motion generation with locality-aware bipartite body-part motion prior,” Computers & Graphics, vol. 136, 2026.DOI
- [12]Jen-Chun Lin, Wen-Li Wei, and Hsin-Min Wang, “Automatic music video generation based on emotion-oriented pseudo song prediction and matching,” in Proceedings of the 24th ACM International Conference on Multimedia (MM), 2016.DOI
- [13]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NIPS), 2017.arxiv.org/abs/1706.03762
- [14]Sander Koelstra, Christian Mühl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras, “DEAP: A database for emotion analysis using physiological signals,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 18–31, 2012.DOI
- [15]Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” in Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980
- [16]Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014.