SemiCodec improves speech tokenization and sound quality with hybrid encoding
SemiCodec: Bridging Speech Tokenization and High-fidelity Reconstruction with Semi-discrete Representations
Sound
Summary
Speech technology often struggles to keep both clear sound quality and easy-to-use data formats. The paper presents SemiCodec, a new approach that uses both discrete chunks and continuous features together to represent speech. The authors develop a special training method to make sure the two parts work well without shortcuts. Their tests show SemiCodec can recreate speech sounds better and keep voices clearer than older all-discrete methods, while matching continuous methods in quality.
What this means in practice
- •For speech technology developers: Develop speech codecs that improve reconstructed audio quality while maintaining clear discrete representations for easier processing.
- •For voice assistant engineers: Enhance voice assistant responsiveness and naturalness by using semi-discrete speech representations balancing fidelity and token efficiency.
Authors
Kecan Mao, Nan Xu, Zhaoyang Meng, Zhengyao Ma, Yingming Gao, Yi Zhao, Ya Li
Abstract
Speech representation learning requires a discrete modeling interface while preserving high waveform fidelity. Existing approaches rarely achieve both: discrete tokens ease modeling but lose fine-grained acoustic information, while continuous latents preserve fidelity but require specialized generative objectives. We introduce SemiCodec, a novel Semi-discrete neural speech Codec with synchronized discrete tokens and continuous latents. Direct joint optimization may create a reconstruction shortcut through the continuous pathway, leaving the discrete tokens insufficiently grounded. To address this issue, we propose a Preserve-then-Ground training paradigm that first establishes a high-fidelity synthesis space and then grounds SSL-guided discrete tokens within it. Experiments show that SemiCodec outperforms fully discrete codecs in reconstruction quality and intelligibility while remaining competitive with continuous representations in acoustic fidelity and speaker preservation. Audio samples are available at https://pennyw1-se.github.io/SemiCodec-Demo-Page/