Speech model learns unified understanding and generation abilities
SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations
Sound
Summary
Understanding and generating speech usually need different kinds of sound information, so most models focus on one task at a time. The authors created SPEAR-Gen, a new model that learns one type of speech representation useful for both understanding what is said and generating speech sounds. They combined techniques that help capture both the meaning in words and the nuances of voice sounds. Tests showed SPEAR-Gen keeps good understanding performance while making speech sounds clearer and more like the original speaker. This means one system could handle both speech recognition and speech generation effectively.
What this means in practice
- •For voice assistant developers: Build voice interfaces that recognize commands accurately and generate natural-sounding speech from the same underlying model.$Commercial implications: This enables companies to produce speech assistants with consistent voice quality and understanding, improving user experience and reducing system complexity.
- •For audiobook producers: Use a single model to both transcribe narration and re-synthesize speech while preserving speaker identity and audio quality.
Authors
Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos, Osama Hanna, Philip C. Woodland, Yiting Lu
Abstract
Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned feature aggregation consolidates complementary linguistic and paralinguistic information across a frozen encoder into discrete targets for masked prediction, while a coarse-to-fine objective combines log-Mel reconstruction with residual flow matching to preserve spectral structure and fine-grained acoustic variation. Experiments on SUPERB and speech resynthesis show that SPEAR-Gen maintains strong understanding performance while substantially improving resynthesis quality and speaker preservation. These results demonstrate that a single speech representation can effectively support both understanding and generation.