Dynamic speech codec boundaries work best at the syllable level
Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries
Sound
Summary
This paper studies how changing the speed of speech encoding by varying segment lengths affects how well machines understand and reconstruct speech. The authors found that the meaning of segment boundaries depends on the type of speech representation used. For models focused on understanding speech content, boundaries that match syllables lead to better reconstruction than those matching phonemes or fixed-length segments. This suggests that when designing speech codecs that vary their frame rates, organizing segments around syllables is most helpful.
What this means in practice
- •For voice assistant developers: Improve speech compression by segmenting audio at syllable boundaries for better reconstruction quality in assistants that rely on semantic speech representations.
- •For telecommunications engineers: Design variable-frame-rate speech codecs that allocate segments based on syllables to reduce distortion in compressed voice communications.
Authors
Han Wang, Jiaqi Li, Yingda Shen, Yuxiang Wang, Zhizheng Wu
Abstract
Dynamic-frame-rate neural speech codecs replace a uniform frame grid with variable-duration tokens, making boundary placement part of the representation itself. Yet it is unclear what these boundaries encode and whether interpretable boundaries are also useful for neural speech reconstruction. This work combines boundary interpretation and controlled reconstruction analysis by comparing predicted boundaries with linguistic and acoustic references. We find that boundary meaning depends on the underlying speech representation. For the ASR-oriented SenseVoice and Whisper encoders, shallow layers emphasize phonetic, voicing, and acoustic transitions, whereas deeper layers shift toward syllable and subword structure. In a comparison of six dynamic-frame-rate algorithms and a uniform (fixed-frame-rate) baseline, higher-level linguistic alignment is associated with lower pooling distortion and better reconstruction from semantic tokens. Frame-rate-matched boundary tests make the distinction concrete: a syllable-derived partition improves over both Uniform and Similarity, while a phoneme-derived partition does not improve over Uniform. We infer that for semantically rich speech representations, useful codec boundaries are best understood as allocation decisions organized around the syllable scale.