ReaFlow TTS improves speech synthesis with mood based control
ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis
Sound
Summary
Making computer-generated speech sound natural is tricky because the same words can be spoken in many different ways. The authors created a system called ReaFlow-TTS that lets the computer learn different ways to say the same phrase and control the mood of the speech by adjusting feelings like excitement or calmness. This system can change pitch, energy, and timing in speech according to these moods, making it more expressive without needing real examples at the time of use. Listening tests showed people could tell the mood changes without losing much naturalness in the voice.
What this means in practice
- •For voice assistant developers: Create speech synthesis systems that generate more natural and emotionally controllable responses for user interactions.
- •For game audio designers: Produce character voices that vary mood and expressiveness dynamically without recording new audio samples.
Authors
Junyi Zhao, Yihao Qin, Changsheng Ma
Abstract
In flow-matching text-to-speech (TTS), different speech realizations can induce different target velocities under the same generation conditions. A deterministic velocity field trained with squared error predicts their conditional mean, thereby marginalizing realization-dependent variation. Meanwhile, modeling such variation does not inherently provide a semantically interpretable interface for attribute manipulation. We propose ReaFlow-TTS, a realization-conditioned flow-matching framework that introduces an utterance-level stochastic realization latent and uses it to condition velocity prediction throughout the generation trajectory. We further impose valence-arousal-dominance (VAD) semantics on the realization space, enabling direct and graded attribute manipulation without target speech at inference. Experiments demonstrate improved synthesis quality over a matched full-mask baseline and reproducible latent-induced pitch, energy, and timing tendencies across initial-noise samples, providing behavioral evidence that the latent is used as a reusable realization condition. Subjective evaluation further demonstrates graded VAD manipulation across generation contexts with only modest changes in naturalness.