Text alone predicts speech style mainly through word choice
Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters
Computation and LanguageArtificial Intelligence
Summary
This paper looks at whether just the words in a sentence can tell us how it should be spoken expressively. The researchers grouped spoken sentences by style using sounds alone, then tried to predict these groups from the text. They found that text predicts speech style somewhat, but mostly because of the specific words used, not deeper text features. This means just text isn’t enough to fully know how something should sound when read aloud.
What this means in practice
- •For text-to-speech developers: Avoid assuming rich prosodic style can be fully inferred from text alone when designing expressive TTS systems.
- •For voice user interface designers: Focus on word choice as a main cue when predicting speaking style from text for conversational agents.
Authors
Abdul Rehman, Jian-Jun Zhang, Xiaosong Yang
Abstract
Much of expressive text-to-speech research rests on an untested assumption that written text carries enough information to select an appropriate prosodic style for its delivery. Text-predicted style models improve listener preference, and expressive-appropriateness evaluation presupposes that context constrains style, yet neither measures the assumption itself. This paper tests it as a falsifiable hypothesis against style labels derived from acoustics alone. For each of six speakers in a 1,200-hour conversational corpus, utterances are clustered in the spaces of five speech models, including a prosody-only control, and the cluster of held-out utterances is predicted from twelve text embedding models. Three controls are applied: utterance length is erased from the speech embeddings; accuracy is scored against the majority-class floor of unbalanced clusters rather than uniform chance; and a bag-of-words baseline measures word identity alone. Text predicts the cluster above that floor for all six speakers (+0.111 top-3 accuracy), but bag-of-words achieves three quarters of this. Sentence embeddings add only +0.026, largest for encoders not trained for sentence semantics and reversed by tree-based probes for all others. Acoustic clusters are not compact in text embedding space in any of 360 configurations. The prosody-only space weakens the association for five speakers, but not for the speaker showing it most strongly. Text thus informs these delivery clusters mainly through word choice, whether as a cue to prosody or as a marker of topic and recording situation, and reference-free style selection cannot assume more.