Autoregressive text to speech models gain on the fly speaking rate control
Controlling Speaking Rate in Autoregressive TTS via Activation Steering
SoundComputation and LanguageMachine Learning
Summary
Text-to-speech systems that create natural-sounding voices usually cannot easily change how fast they talk after they are trained. The authors found a way to adjust the speaking speed during speech generation by modifying a specific internal part of the model. They identify a direction in the model’s activations related to speech rate, allowing them to clamp and control this speed without retraining. This method keeps the speaker's unique voice and sounds natural, and it works well across different models and in tests with humans.
What this means in practice
- •For voice assistant developers: Adjust speaking pace dynamically in voice assistants without retraining speech models for different rates.$Commercial implications: Enables customizable voice speed in assistants which can be marketed to enhance user experience.
- •For game audio engineers: Control character speech speed in games in real time to match gameplay pacing without multiple voice samples.
Authors
Francesco Verdini, Antonis Asonitis, Aref Farhadipour, Marzieh Razavi, Pierre-Edouard Honnet, Vijeta Avijeet, Juan Pablo Zuluaga Gomez
Abstract
Autoregressive text-to-speech (TTS) systems synthesize natural speech but, once trained, offer little control over speaking rate. We show that speaking rate can be steered at inference time, without retraining, by clamping a single decoder block's activation along a discovered speed axis. A decoder-block analysis recovers the rate axis, a neutral operating point, and a per-step intensity scale; at inference, the activation's projection onto this axis is set to a fixed scalar. Learning this direction from synthetically time-stretched and time-compressed speech yields rate control that largely preserves speaker identity, generalizes across model architectures, and maintains high naturalness in objective and human evaluations. Unlike standard additive steering, which breaks at the slow extreme, clamping remains stable on all three systems tested; at moderate targets, the better rule depends on the model. Finally, we show that rate information is decodable across layers but causally steerable only within a mid-depth window, and demonstrate the effectiveness of our approach on the public Seed-TTS-Eval benchmark.