High decodability in speech models does not ensure precise speaker control
Decodability Is Not Controllability: Local and Anisotropic Speaker Control in Continuous-Latent TTS
SoundComputation and LanguageMachine Learning
Summary
This paper shows that just because a voice model can easily decode speaker identity, it does not mean one can accurately change or control the speaker's voice as intended. The authors found that edits made at different points inside the model affect voice changes very differently—some do not result in the target speaker's voice being heard at all. Their experiments confirm that truly controlling speaker identity in text-to-speech requires targeting very specific parts of the model and ensuring the speaker information is correctly passed through the system.
What this means in practice
- •For voice assistant developers: Use geometrically precise edits in speech synthesis to reliably change a voice to a target speaker without retraining the model.
- •For audio forensics teams: Improve techniques to verify speaker identity changes in synthesized speech by focusing on causal model interventions rather than decoder accuracy alone.
Authors
Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang
Abstract
High linear decodability does not entail causal controllability in continuous-latent text-to-speech (TTS). In VoxCPM2, position-matched probes and equal-norm interventions across 30 speakers and 60 directed dyads expose this separation: reference-audio states at the final residual acoustic language model layer (RALM L7) retain 0.970 probe accuracy, yet aligned edits yield 0/60 target-dominant dyads. Edits at early reference-audio positions produce the opposite pattern. At L0, pair-aligned updates increase target similarity by 0.326 and achieve target dominance in all 60 dyads, against at most 1/60 under any equal-norm direction or position control. Independent speaker encoders, retrieval against held-out speaker prototypes, and double-blind listening confirm that the shifted voice is heard as the intended target. Cross-architecture interventions in dots.tts distinguish backbone leverage from direct acoustic control: backbone edits displace the source without target attraction, while x-vector ablation reduces source similarity from 0.849 to 0.006. These findings identify two requirements for training-free speaker control: geometric specificity and a computational route that carries the intended speaker condition into acoustic synthesis.