Tracing key components controlling emotion in AI speech synthesis
Tracing a Sparse Emotion-Control Circuit in LLM-Based Text-to-Speech
SoundArtificial Intelligence
Summary
Text-to-speech models can add emotions to computer-generated speech, but it's unclear how these emotions work inside the models. The authors developed new ways to measure emotion signals and tested which parts of the model control emotional expression. They found a small set of components responsible for creating emotional changes in speech and showed these cause real acoustic differences like pitch and tone. This helps us understand how emotions are turned into voice in AI systems.
What this means in practice
- •For voice technology developers: Identify precise model components to improve emotional expression in AI speech synthesis systems.
- •For audio software engineers: Develop targeted interventions in speech synthesis to modify emotion-related voice properties like pitch or energy.
Authors
Hongfei Du, Jiacheng Shi, Yanfu Zhang, Ye Gao
Abstract
LLM-based text-to-speech (TTS) models can generate emotionally expressive speech, but how reference emotion is routed through the model and realized in decoded speech remains unclear. We introduce two emotion-sensitive metrics for matched neutral and emotional syntheses---a codec trajectory score and a late residual direction score---and use them to score activation-patching interventions. Under controlled matched-reference conditions, this analysis identifies a sparse source-to-readout component-level circuit: 23--27 attention heads and MLPs per emotion, roughly 5% of the components considered, recover or suppress 74--88% of the late emotion-readout shift on held-out cases. The circuit combines a shared component backbone with emotion-specific components; cross-emotion activation swaps reduce the target readout in 47 of 48 cases. In decoded speech, the same intervention produces consistent changes in pitch, energy, and spectral brightness over 24 matched pairs per emotion. A readout-matched residual-direction baseline produces only 17--27% of the intervention's pitch effect, showing that internal readout movement alone does not explain the decoded acoustic changes. These results trace a compact causal route from reference-derived prefix information to emotion-relevant properties of generated speech.