Continuous-time modeling improves speech emotion tracking in text to speech
Continuous-Time Acoustic Modelling with Neural Controlled Differential Equations
SoundArtificial Intelligence
Summary
When computers turn text into spoken words, they need to decide how long to say each part, like syllables or sounds. Usually, these lengths only change the timing but not the details of how the speech sounds. The authors used a new math tool called neural controlled differential equations to model the speech sounds continuously over time. This lets the speech better capture emotions and timing simultaneously, making the computer voices sound more expressive and natural. They showed that this method can track emotional intensity more accurately without losing quality.
What this means in practice
- •For text-to-speech developers: Create TTS voices that better capture emotional intensity by modeling speech timing continuously rather than in fixed steps.$Commercial implications: Enables more natural and style-expressive synthetic voices for virtual assistants and audiobooks, improving user engagement.
- •For signal processing engineers: Improve acoustic feature extraction by using continuous-time representations to better align speech timing and phonetic content.
Authors
Mattias Cross, Minghui Zhao, Anton Ragni
Abstract
Text-to-speech (TTS) models commonly address text--speech alignment by expanding phone-level encoder states to frame-level decoder inputs using predicted durations. While this length-regulation step resolves alignment structurally, this use of duration typically changes only where and how often latent states appear, not the values of the states themselves. This paper proposes a continuous-time mechanism for duration-aware acoustic modelling in TTS using neural controlled differential equations (CDEs). We formulate the phone representation as a temporally parameterised control path and use a neural acoustic vector field to produce a continuous-time hidden state whose values evolve with phonetic content and duration-derived timing. The resulting trajectory can be sampled at discrete points and integrated into a standard acoustic decoder pipeline. Objective results contrast CDEs and typical recurrent models. Subjective results suggest that CDE-based models evaluating one phone per step can improve rank-order agreement between synthesised and reference emotion intensity while maintaining comparable emotion-expression quality to a strong baseline. Additional experiments with half-phone step-sizes suggest that temporal resolution changes the trade-off between style tracking and absolute calibration. These results position CDEs as a promising design space for continuous-time and duration-aware style-sensitive TTS.