SAGE autoencoder balances audio quality semantic meaning and speed
SAGE: Semantic Audio Generative Encoder
SoundMachine Learning
Summary
Audio autoencoders compress sounds into small, meaningful codes that other programs can use. The authors created SAGE, a new audio autoencoder that keeps sounds clear, organizes these codes in a way that captures meaning, and works fast. They trained SAGE on publicly available music and tested it against bigger, slower models, finding it performs better or just as well while being much faster. This work helps make audio processing more efficient without losing quality or useful information.
What this means in practice
- •For music technology developers: Encode high-quality music audio into small, meaningful codes quickly for downstream audio generation or editing tools.
- •For mobile app developers: Build apps that analyze or transform music audio efficiently without sacrificing sound quality or semantic understanding.
Authors
Francesco Brigante, Luca Cerovaz, Davide Marincione, Giorgio Strano, Luca Zhou, Emanuele Rodolà, Michele Mancusi
Abstract
Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs at the inference cost of Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and 4x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.