Summary
Speech compression systems turn sounds into short pieces called tokens so computers can store or recreate speech. Usually, shorter token sequences mean each token carries more info, making quality tricky. The authors present RubatoCodec, which adjusts token lengths based on speech changes but keeps an exact, fixed number of tokens per utterance, helping speech generators handle streams one event at a time. Tests show RubatoCodec keeps better speech quality and understanding than fixed-rate methods, works well in multiple languages, and improves automatic speech recognition when used for text-to-speech. This approach supports both high-quality speech coding and continuous speech generation.
What this means in practice
- •For speech technology developers: Build adaptive speech coding systems that control token counts exactly to improve speech quality and decoding latency.
- •For text-to-speech developers: Improve streaming text-to-speech quality and automatic speech recognition accuracy using fixed-budget adaptive token representations.
- •For communication platform engineers: Integrate bandwidth-efficient speech codecs with controlled token use for enhanced streaming voice applications.$Commercial implications: Enables adaptive, budgeted speech compression products for streaming communication services that deliver quality under fixed bandwidth.
Authors
Chihiro Arata, Shohei Taniguchi, Gouki Minegishi, Kan Hatakeyama-Sato
Abstract
Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.