Zero shot text to speech controls speaker identity and accent separately
DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Sound
Summary
Creating computer voices that sound like a specific person and speak with a particular accent can be tough because these features are usually mixed together. The authors present DEFINE, a system that separates a person’s voice identity from their accent using different audio examples. This lets the system change the strength of an accent without needing extra training. It works well even on accents it hasn’t heard before, keeping the speaker’s voice consistent while changing how the accent sounds.
What this means in practice
- •For voice assistant developers: Generate personalized voices that separately adjust speaker identity and accent from short audio samples without retraining.$Commercial implications: This enables building customizable virtual assistants with flexible accent control appealing to diverse users.
- •For language learning app developers: Produce speech samples that independently manipulate speaker voice and accent to expose learners to varied pronunciations.
Authors
Ambuj Mehrish, Abhinaba Roy, Alex Ivanov, Tawsif Ahmed, Dorien Herremans
Abstract
Zero-shot text-to-speech (TTS) can reproduce an unseen speaker from a short reference recording, but typically entangles speaker identity and accent within the same reference. We introduce DEFINE, an end-to-end framework that decouples these factors by conditioning speaker identity and target accent on separate audio exemplars. A single inference-time guidance weight continuously controls accent strength without retraining. Built on F5-TTS with parameter-efficient LoRA adaptation, DEFINE maps short accent exemplars into a conditioning space using an exemplar encoder supervised through learned accent prototypes, requiring neither accent labels at inference time nor post-synthesis waveform conversion. On seen accents, increasing accent guidance improves accent-probe accuracy from 6.5% to 19.6%. More importantly, a single DEFINE model generalizes accent control beyond its training accent set: on seen and out-of-domain accents, though not on held-out accents, it matches the accent transfer performance of a two-model TTS-voice-conversion cascade while achieving higher speaker similarity and comparable predicted speech quality. These results demonstrate that speaker identity and accent can be independently controlled from audio exemplars within a single zero-shot TTS model, including for accents unseen during training.