Speech encoder learns multiple voice traits at once with multi-teacher method
Learning Speech Attribute Representations via Multi-Teacher Distillation
Computation and Language
Summary
Speech systems usually learn one thing at a time, like understanding words or recognizing the speaker. This paper explores whether one system can learn many voice features at once, such as what is said, who is speaking, the language, and the emotion. The authors use a method where a main system learns from four experts specializing in each trait simultaneously. The result is a single speech encoder that performs nearly as well as systems trained only for one task, but much more efficiently. They also found that different voice traits are best recognized at different levels inside the system, offering clues about how speech information is organized.
What this means in practice
- •For speech technology engineers: Build more efficient voice recognition systems that can handle speaker identity, language, emotion, and meaning in one model.
- •For customer service platform developers: Develop call analysis tools that simultaneously detect caller language, emotion, and identity with a single speech encoder.$Commercial implications: Enables a multifunctional call monitoring product for service centers by combining key speech attributes in one model.
Authors
Maryem Bouziane, Salima Mdhaffar, Yannick Estève
Abstract
Recent post-training approaches have shown that self-supervised speech encoders can be adapted to learn a single utterance-level speech attribute through knowledge distillation. In this work, we investigate whether a single speech encoder can simultaneously preserve multiple complementary utterance-level speech representations. We propose a multi-attribute multi-teacher distillation framework that learns semantic, speaker, language, and emotion representations from four frozen teacher models using a shared SSL backbone and attribute-specific branches. Experiments on multilingual semantic retrieval, speaker verification, language identification, and speech emotion recognition show that the proposed model achieves performance close to dedicated single-task students while requiring only a single shared encoder. Furthermore, the learned layer aggregation strategies reveal that different speech attributes naturally emerge at different depths of the SSL encoder, providing new insights into the organization of speech representations.