ZipCodec compresses speech using ultra-low frame rate and bitrate
ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
SoundArtificial IntelligenceMachine Learning
Summary
Compressing speech for streaming usually requires sending many small pieces quickly, which is hard to do well at very low frame rates. The authors present ZipCodec, a new technology that sends fewer pieces of speech data per second while keeping sound quality good enough to understand. They built ZipCodec using advanced methods like a special transformer design and clever data compression techniques. ZipCodec works fast enough to run in real time on regular computers and is better than other tools at similar data rates.
What this means in practice
- •For telecommunications engineers: Integrate ZipCodec to enable low-latency, low-bitrate voice streaming with reduced data usage in live communication systems.
- •For speech recognition developers: Use ZipCodec’s compressed speech streams to improve accuracy in downstream speech processing tasks under constrained bandwidth.
- •For assistive technology builders: Implement ZipCodec for real-time speech transmission in devices aiding people with speech or hearing impairments to lower computational demands.$Commercial implications: Enables product features for assistive communication devices that require efficient, low-latency speech coding operable on standard hardware.
Authors
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Abstract
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.