CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
2026-06-03 • Sound
SoundComputation and Language
AI summaryⓘ
The authors developed CleanCodec, a new audio compression method that focuses on keeping only the important parts of speech sounds and ignoring background noise or unwanted artifacts. This approach treats audio encoding as a problem of selecting useful information while filtering out the rest. CleanCodec uses fewer tokens per second to represent audio, leading to better speaker similarity and clearer speech than previous methods. It also makes speech-related tasks like text-to-speech and voice conversion faster and more efficient.
neural audio codecsaudio tokenizationinformation bottleneckdenoisingspeech intelligibilityspeaker similaritytext-to-speechvoice conversiontoken efficiencyinference speed
Authors
Eugene Kwek, Feng Liu, Rui Zhang, Wenpeng Yin
Abstract
Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.