Voice cloning models reveal interpretable speech features for better control
Discovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS Models
Computation and LanguageMachine LearningSound
Summary
Voice cloning technology can copy a person’s voice from short speech clips, but it’s unclear how these systems understand different sound traits. The authors studied several popular voice cloning models to find out which voice qualities like pitch, accent, and speaking style are captured and how easy it is to identify or change these traits. They used special methods to discover interpretable speech features and found that some methods work better for spotting traits, while others help more with changing them. Their work helps explain what voice cloning models learn about speech and could make it easier to control synthetic voices in the future.
What this means in practice
- •For speech technology developers: Identify and modify specific speech traits in voice cloning systems to improve customization and quality of synthetic voices.
- •For audio forensics teams: Use interpretable acoustic features uncovered in voice cloning models to analyze speaker identity, speaking style, or recording conditions in forensic audio.
Authors
Anuj Diwan, David Harwath, Eunsol Choi
Abstract
Voice-cloning TTS models synthesize an unseen speaker's voice from brief speech prompts, but it remains unclear which acoustic properties they represent and use. We introduce a feature-based interpretability framework for autoregressive voice-cloning TTS models, analyzing features from sparse autoencoders (SAEs), PCA, and random directions via sparse probing, steering, and quantitative auto-interpretability. We focus on Qwen3-TTS and extend probing and steering to $5$ additional models to test generalization across architectures. These models encode concepts like pitch, energy, noise, and accent, but sparse prediction and steering are effective only for some concepts. Strong probing does not always translate to steering or vice-versa. SAEs are useful for sparse probing of human-labelled concepts and novel-concept discovery, while PCA is competitive for dense probing and steering. Auto-interpretability finds $684$ automatically validated SAE features in Qwen3-TTS covering speaker clusters, speaking style, and recording conditions beyond known concepts. Overall, we perform a comprehensive interpretability study of autoregressive voice-cloning TTS models.