Audio encoders keep different amounts of sound details in embeddings
How Much Audio Is Left In An Embedding? An Inversion Audit Of Audio Encoders
SoundArtificial Intelligence
Summary
This paper looks at how much original music information remains inside different types of audio encoders after they convert sound into compact codes for other tasks. The authors test reconstructing short music clips from these codes to see how well details like timing and pitch survive. They find that some encoders keep more of the sound’s specifics than others, and that exposing finer details in the codes helps make better reconstructions. Even compressed audio representations still hold enough information to recover recognizable musical content.
What this means in practice
- •For audio software developers: Use insights on encoder detail retention to choose embeddings for better music reconstruction from compressed codes in apps.
- •For music streaming platform engineers: Assess which pretrained audio embeddings preserve enough music detail for enhanced downstream tasks like recommendation or analysis.
Authors
Marios Glytsos, Brian McFee
Abstract
Pretrained audio encoders are reused for downstream tasks that are often unknown when the encoder is trained, so their usefulness depends partly on which signal properties survive the pretext objective. We study this retained information through paired source reconstruction. Using a shared Stable Audio Open latent diffusion decoder, we reconstruct five-second, 44.1-kHz stereo music from frozen representations produced by supervised classifiers (VGGish, ConvNeXt), an audio-text contrastive model (CLAP), and a waveform-reconstruction model (EnCodec). These objectives impose different pressures to preserve source detail, while their exposed interfaces vary substantially in temporal and spectral resolution. Evaluating on the Million Song Dataset (MSD), we find clear differences in reconstructability across encoder families, while within encoder comparisons show improved recovery when finer temporal or spectral structure is exposed. Even compressed task oriented embeddings support reconstructions that preserve measurable source specificity and high level musical content.