Streaming Neural Speech Codecs through Time-Invariant Representations
2026-07-06 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study a new way to break down speech into parts that change over time and parts that stay the same, using a method called TIRE in their TiCodec system. They find that different layers of the speech encoder capture different types of information like speaker identity and environment noise but not much about the actual words spoken. They improve the system by combining information from multiple layers, calling this Dual-TIRE, which helps make better speech reconstructions that sound more like the original speaker. They also show their method works well when processing speech in short chunks, making it suitable for real-time applications.
neural speech codecfactorized representationTime-Invariant Representation Extraction (TIRE)encoder layersspeaker informationlinguistic contentcross-file samplingDual-TIREspeech reconstructionstreaming inference
Authors
Kélian Estève, Salima Mhdaffar, Mickael Rouvier, Richard Dufour, Yannick Estève
Abstract
Neural speech codecs are increasingly used as intermediate representations in codec-based speech generation systems. TiCodec introduces a factorized representation that separates time-varying speech content from time-invariant information through a Time-Invariant Representation Extraction (TIRE) module, potentially reducing the amount of information that must be modeled at the frame-level. In this work, we investigate the nature of the information captured by TIRE representations and their suitability for low-latency speech processing. Using a series of probing tasks, we analyze the influence of the encoder layer and show that intermediate layers capture complementary speaker- and environment-related information while containing little linguistic content. We further study several segment selection strategies for TIRE training and demonstrate that cross-file sampling improves the robustness of invariant representations. Based on these findings, we propose Dual-TIRE, a multi-level architecture that exploits the complementarity of different encoder layers and improves speech reconstruction quality and speaker similarity. Finally, we evaluate TiCodec in a streaming inference setting using successive 660ms processing blocks. Results show that streaming operation can be achieved without significant degradation in reconstruction performance, highlighting the potential of factorized neural codec representations for future low-latency speech generation systems.