Neural audio codecs encode speaker traits at different depths
Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics
Sound
Summary
Neural audio codecs compress speech into codebooks but it was unclear how well these codes captured different speaker traits like sex, age, or emotion. The authors tested 17 codecs to see which traits are easier or harder to detect from parts of the codec representations. They found some traits like speaker sex are captured in simpler parts of the code, while others like voice quality need more detailed codebooks. This helps understand how audio codecs work and why they might need different designs for certain speech features.
What this means in practice
- •For audio codec engineers: Design neural audio codecs that target specific speech traits by understanding where these traits are encoded in codebook layers.
- •For speech technology developers: Evaluate and improve speech language models by selecting codec codebooks that best preserve desired speaker traits.
Authors
Jihwan Lee, Tiantian Feng, Minu Kim, Junhyeok Lee, Sean Foley, Thanathai Lertpetchpun, Najim Dehak, Dani Byrd, Shrikanth Narayanan
Abstract
Neural audio codecs are increasingly used for low-bitrate speech coding and serve as tokenizers for speech language models, yet it remains unclear how their codebook hierarchies encode speaker traits and speech characteristics. We systematically study 17 multi-codebook neural audio codecs across seven traits: sex, age, accent, voice quality, emotion, speech flow, and non-verbal sounds. For each codec, we reconstruct speech using partial codebooks and use machine listeners to compare trait predictions on each reconstruction. We measure prediction agreement with reweighted Cohen's \k{appa}, and define trait depth as a proxy for the codebook depth by which each trait is encoded and recoverable. Overall, sex has the shallowest trait depth, followed by speech flow, whereas voice quality and age group require deeper stacks; emotion, accent, and non-verbal sounds fall in between. Within each trait, speech-flow classes are at relatively similar trait depths, whereas emotion, accent, and non-verbal sounds differ more. Additionally, we explore how certain codec design choices relate to trait depths. These findings show that codebook hierarchies are trait-dependent and may not exhibit uniform coarse-to-fine structures, motivating trait-specific codec evaluation and design for future applications.