Neural audio codec tokens show strong link to specific speech sounds
Probing the phon* representations of the Mimi codebooks
Computation and Language
Summary
Neural audio codecs compress speech into short tokens for use with language models. This study found that the first group of tokens in the Mimi codec strongly relate to particular speech sounds (phonemes), though each phoneme can be represented by several different tokens. The researchers used methods better suited to Mimi’s long token length instead of usual tests. Their findings reveal the complexity of how speech sounds are encoded in neural audio compression.
What this means in practice
- •For speech technology developers: Improve speech compression systems by understanding how phonemes map to codec tokens for better integration with language models.
- •For voice assistant developers: Use insights on token-phoneme associations to enhance speech recognition and synthesis quality in constrained audio encoding scenarios.
Tested on one dataset.
Authors
Artem Saloev, Nicolas Ballier, Erin Pacquetet
Abstract
Neural audio codecs are increasingly used to encode speech for integration with large language models. This paper investigates the phonetic representations in the first codebook (CB0) of the Mimi codec. ABX style evaluation is poorly suited to Mimi fixed 80 ms tokenization window, which often spans multiple phonemes; we instead use forced alignment, entropy analysis, and Phone-Normalized Mutual Information (PNMI). Using TIMIT speech (58/55 speakers, SA1/SA2), individual tokens are strongly associated with specific phonemes (PNMI = 0.84, above random), yet phonemes are realized by multiple competing tokens rather than a one-to-one mapping.