Structural Bottlenecks on Frequency Representation in End-to-End Audio Models
2026-07-09 • Sound
SoundMachine Learning
AI summaryⓘ
The authors studied how certain neural audio models compress sounds and found that the models often mix together important details like pitch and timbre, making it hard to separate these features. They showed that this happens because of structural limits in the common convolutional encoder designs. To fix this, they created a method called Gabor Latent Refactorization (GLRF) that reorganizes the model’s internal representations without retraining, improving how well these features can be distinguished and controlled. Their work suggests that while current models lose some detail, this can be partly recovered with a simple adjustment.
neural audio modelspitchtimbrestrided convolutional encoderstime-frequency primitivesaliasingfrequency resolutionGabor Latent Refactorizationlatent representationssignal processing
Authors
Nicole Cosme-Clifford
Abstract
End-to-end neural audio models achieve high-fidelity compression and generation. We might read that performance as evidence they directly represent interpretable features such as pitch and timbre, but a model can produce plausible outputs without doing so. A model may encode these features in any reachable basis, but regardless of which, the features are well described as compositions of time-frequency-localized primitives. Whether state-of-the-art encoders preserve access to these primitives, and thus to compositions of them, remains unclear. Through theoretical analysis and controlled experiments, we show that several state-of-the-art strided convolutional encoders impose two structural bottlenecks, both predictable from architecture and signal structure, on access to these primitives: (1) they collapse primitives into alias equivalence classes, establishing a bound on representational capacity, and (2) they limit the frequency resolution available to learned filters, restricting separability. For well structured data, we find collapse rates of 31-35% and filter bandwidths 10-35x above the theoretical resolution bound, confirming that both bottlenecks arise under realistic signal conditions. We then introduce Gabor Latent Refactorization (GLRF), a lightweight post-hoc intervention that re-expresses encoder latents in a frequency-localized basis, reducing filter bandwidths from 10-35x to 1.5-3x of the theoretical resolution bound while preserving reconstruction fidelity and improving control over attributes like pitch. These results show that the encoders in question predictably degrade access to frequency-localized primitives, entangling the features that depend on them, and that a lightweight, retraining-free intervention can recover much of that access, improving steerability and interpretability.