Audio representations partly capture additive sound compositions
Do Audio Representations Compose Additively?
SoundMultimedia
Summary
This paper asks whether modern audio models can understand complex sounds as sums of simpler parts, like hearing a song made of different instruments. The researchers tested three popular audio models to see if their internal sound features combine linearly when sounds mix together. They found that all models show some ability to add simpler sound features to represent combined sounds, but only one model trained with audio and text together showed strong additive patterns. Still, none perfectly captured sound mixes, showing limits to these models’ understanding of complex sound structures.
What this means in practice
- •For audio software developers: Improve sound separation tools by using tested models that partly represent sounds as additive combinations of source features.
- •For speech technology teams: Understand limits of current speech models in combining multiple sound sources, avoiding overreliance on additive assumptions.
Authors
Chenhao Xue, Zhijin Guo, Joyraj Chakraborty, Martin Reed, Nikolaos Thomos
Abstract
Compositionality, the ability to represent complex acoustic scenes as combinations of simpler sound sources, is central to auditory perception and classical additive signal models. Still, it remains unclear whether modern pre-trained audio representations internalize additive structure without compositional supervision. Existing evaluation frameworks of audio compositional reasoning largely focus on cross-modal audio-text alignment, leaving open whether audio representations themselves exhibit additive compositional structure independent of text grounding, analogous to vector arithmetic in word representations. To investigate this, we adopt a two-step diagnostic for frozen audio representations. First, we quantify linear alignment between representations and sound source labels using canonical correlation analysis. Second, we test additive compositional generalization via leave-one-combination-out reconstruction, grouping clips by exact source-label set, averaging their representations, and predicting held-out means from per-source contributions fitted only on training combinations. With larger combination holdouts, CLAP outperforms the permuted and label-overlap baselines on FSD50K, while the speech models do not outperform the label-overlap baseline. We examine representations generated by Wav2Vec2, HuBERT, and CLAP on FSD50K and CHiME-Home datasets. All three models show consistently higher linear correlation and more accurate leave-one-combination-out reconstructions than the permuted baselines. However, only CLAP shows large cosine similarity gains, which could be associated with its training on many kinds of audio and text. Finally, we note that all three models exhibit reconstruction residuals, revealing limits of additive compositionality such as nonlinear or non-compositional audio structure.