Neural audio decoders show layered encoding of musical features
Feature Encoding in VAE-based Audio Decoders: Effects of Input, Depth and Distribution
SoundMachine Learning
Summary
Neural audio models can create music sounds, but how they represent musical features inside hasn't been clear. The authors studied a popular audio model called RAVE and found that its internal layers encode things like pitch and tempo, especially when using synthetic sounds. Real music sounds also show this encoding, but it's weaker unless more complex analysis is used. These patterns are seen across different models and could help improve how music AI systems are controlled and understood.
What this means in practice
- •For audio software developers: Use knowledge of where musical features are encoded in audio decoders to design better controls for AI music synthesis tools.
- •For music production companies: Develop more interpretable AI audio synthesis products by leveraging insights into feature encoding patterns across neural network layers.$Commercial implications: Enables selling advanced neural audio synthesis tools with improved user control and transparency in music production.
Authors
Louis McCallum, Mick Grierson
Abstract
Neural audio synthesis models like the Realtime Audio Variational autoEncoder (RAVE) achieve impressive genera tion quality, yet how their internal representations encode musical features remains poorly understood. We present a systematic layer-wise and cross-layer cluster analysis of RAVE decoder activations across three models trained on different musical domains, tested with four stimulus types. We then evaluate architectural generalization with a general purpose EnCodec model. For RAVE, we find that synthetic stimuli are encoded well across models and audio features (pitch |\r{ho}|=0.45, 5.1x the null, BPM |\r{ho}| = 0.76, 8.6x the null). These results are reduced but still substantively apparent when using natural audio (mean across features |\r{ho}|=0.25, 2.8x the null). Natural audio sees a stronger encoding when nonlinear probes are used (mean across features R2=0.56, 18x the null, +0.152 nonlinear gain over the linear probe R2). Encoding strength varies throughout the layers of the decoder and an increased ability to joint-encode in the middle layers is seen across all audio features (\b{eta}2 all negative, p < 0.05). The general purpose EnCodec decoder also sees similar strong synthetic responses across audio features, similar nonlinear gains for natural audio joint encoding and similar depth profiles. We find the best cross-layer cluster improves the strength (r = 0.65, p = 0.006) and prevalence (r = 0.75, p = 0.001) of BPM encoding when compared against the best whole layers within the same section, with no effect for joint encoding. These findings advance the interpretability of neural audio models and inform targeted control strategies for neural synthesis.