Speech text models improve semantic coherence but differ in phonetic and speaker quality
Quantifying the Generation Modality Gap in Speech-Text Language Models
Computation and Language
Summary
It is harder for speech-only AI systems to generate meaningful and smooth content compared to systems that use both speech and text. The authors measured these differences by training and testing speech-only, text-only, and combined speech-text models on similar data and tasks. They found that combining speech and text helps the AI better understand and continue the meaning of speech, but some features like the details of sounds and speaker consistency don’t improve as much. This shows that using text data helps AI learn language meaning more efficiently when dealing with spoken language.
What this means in practice
- •For voice assistant developers: Improve speech generation quality by training models combining speech and text data to produce more coherent spoken responses.
- •For call center technology teams: Use insights on emotion consistency and speaker quality metrics to enhance the naturalness and emotional tone of AI-generated speech in customer interactions.
Authors
Ju-Chieh Chou, Jiawei Zhou, Karen Livescu
Abstract
Pure speech language models often lag behind text and speech-text language models in generating coherent content, but this gap is difficult to quantify because speech and text systems are typically evaluated with different metrics and trained on different data. We study the speech-text modality gap in a family of spoken language models, based on flow matching for continuous acoustic feature generation. We construct a unified generation-based evaluation suite that compares speech-only, text-only, and speech-text language models trained on matched data distributions and evaluated in matched generation settings. We evaluate generated continuations along multiple dimensions: semantic coherence, measured by transcribing generated speech and scoring it with a reference language model; local phonetic structure, measured by phone n-gram distributional statistics; speaker consistency and acoustic quality; and emotion-based distributional metrics. Across datasets, we find that joint speech-text modeling substantially improves semantic coherence. However, the improvement is not uniform across metrics: phone-level metrics change only modestly, speaker similarity and predicted quality are lower for speech-text continuations, while emotion-based distributional metrics improve. Compared with larger-scale speech-only models, our speech-text model closes much of the scaling gap in transcript-based semantic coherence, suggesting that text provides an efficient semantic training signal for spoken language modeling.