Face images control emotion intensity in expressive text to speech
EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Sound
Summary
Expressing emotions at different intensities is hard for text-to-speech systems, especially when the system doesn't have explicit emotion labels to guide it. The authors propose a model called EmoFI-TTS that uses a single face image to figure out both the type and strength of an emotion, while also extracting who is speaking from an audio sample. They combine these clues to generate speech that sounds both emotional and natural, and let users control how strong the emotion is without retraining the model. Their tests showed that EmoFI-TTS accurately matches the emotion intensity and keeps the speech understandable and natural sounding.
What this means in practice
- •For voice assistant developers: Generate expressive voice responses based on user's facial emotions to improve interaction naturalness without needing explicit emotion labels.
- •For game audio designers: Create character speech that varies emotional intensity dynamically by using character face images as input.
Authors
Yejin Jeon, Luel Hagos Beyene, Marie Maltais, Hyounghun Kim, David Ifeoluwa Adelani
Abstract
Controlling emotional intensity remains challenging in expressive text-to-speech synthesis, particularly when emotion must be inferred without explicit labels. We propose EmoFI-TTS, a face-conditioned framework that derives emotion category and intensity from a single facial image while extracting speaker characteristics from reference audio. Modality-specific representations are fused to condition a pretrained speech synthesizer, with cross-speaker training encouraging separation of affect and vocal identity. An intensity-conditioned refinement module enhances the generated mel-spectrogram, while interpolation of learned intensity embeddings enables continuous control without additional training. On speaker-disjoint MEAD evaluation, EmoFI-TTS achieves on average 64.4% and 75.2% objective and subjective emotion intensity accuracies, while retaining low word error rate and high naturalness.