Steerspeech lets text to speech voices express emotion with precision
Steerspeech: Activation Steering For Emotion Control In Generated Speech
SoundMachine Learning
Summary
Text-to-speech systems can speak with emotion, but controlling the feeling consistently and clearly is hard. The authors present SteerSpeech, a way to finely adjust emotions like happiness or sadness in speech by tweaking the model’s internal signals without retraining everything. This method works quickly and keeps the voice and words clear. It improves emotional expression across different voices and languages compared to earlier methods.
What this means in practice
- •For voice assistant developers: Integrate emotion control into text-to-speech voices to improve user engagement with natural and variable emotional speech.
- •For multimedia content creators: Produce speech audio with customizable emotional tones for characters or narration without extra model training.
- •For audiobook production teams: Generate expressive readings with controlled emotional intensity, enhancing listener experience through fine emotion tuning.$Commercial implications: Enables audiobook companies to create tailored emotional speech narrations with minimal customization effort.
Authors
Afsara Benazir, Darius Pétermann, Felix Xiaozhu Lin, Salar Rahili
Abstract
Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.