Transformers improve audio quality by restoring missing high frequencies
Multi-Rate Bandwidth Extension by Token Completion in Neural Audio Codecs
Sound
Summary
Sounds often lose their high-pitched details when compressed or transmitted, making them less clear. The authors view this as a puzzle where a computer predicts the missing sound pieces using a transformer, a type of AI model. They designed a new sound encoding system that helps the AI better guess these missing parts. This approach results in clearer sounds that people and machines agree are better. It shows that designing the sound encoding and AI prediction together leads to improved audio restoration.
What this means in practice
- •For audio streaming services: Restore high-frequency sound details lost during compression to improve user listening experience in streaming apps.$Commercial implications: Enables streaming services to offer clearer audio by upgrading codecs, enhancing product quality and customer satisfaction.
- •For hearing aid manufacturers: Enhance hearing aids by reconstructing lost high-frequency sound components for clearer, more natural audio perception.$Commercial implications: Allows hearing aid products to deliver improved sound clarity through advanced neural processing techniques.
Authors
Benoît Ginies, Olivier Fercoq, Gaël Richard
Abstract
Bandwidth extension, the task of reconstructing the high-frequency components of an audio signal from its low-passed counterpart, is a long-standing problem in audio processing. In this work, we extend recent advances in neural architectures by framing bandwidth extension as an audio token prediction problem. Specifically, we train a transformer-based language model on the discrete representations produced by a disentangled neural audio codec, where the disentanglement is guided by a Harmonic-Percussive decomposition of the input signals, highlighting spectral structures particularly relevant for bandwidth extension. Our approach introduces a novel codec design that explicitly accounts for the downstream token prediction task, enabling a more effective coupling between codec structure and transformer modeling. This joint design yields high-quality reconstructions of the original signal, as measured by both objective metrics and subjective evaluations. These results highlight the importance of aligning codec disentanglement and representation learning with the generative modeling stage, and demonstrate the potential of global, representation-aware design for advancing bandwidth extension.