Tone metric for multilingual text to speech improves tonal language accuracy
Tone on a Budget: A Reference-Free Metric for Lexical Tone in Massively Multilingual Text-to-Speech
Computation and Language
Summary
Text-to-speech systems struggle to capture the correct tones in languages like Yorùbá, where pitch changes the meaning of words. Existing methods that measure errors miss tone mistakes because they ignore tone marks in writing. The authors created DunDun, a new way to check if synthesized speech matches the expected tones using only the pitch in the audio and the text’s tone marks. They showed DunDun works better than traditional error rates and tested it with human listeners and modifications to the audio. The method also reveals tone quality in multiple languages without needing tone-labeled speech data.
What this means in practice
- •For speech synthesis developers: Evaluate and improve tonal accuracy in multilingual text-to-speech systems without needing tone-labelled speech data.
- •For language technology integrators: Measure tone quality automatically in tonal and non-tonal languages to better tune and benchmark speech models.
Authors
Moses Daudu, Adeola Enitan Bamidele, Honor-Jesus Bezaleel
Abstract
In Yorùbá, pitch alone separates \d{o}k\d{o} (husband, Mid), \d{o}k\d{ò} (vehicle, Low), and \d{o}k\d{ó} (hoe, High) -- the diacritics ARE the tone marks. Yet character error rate (CER), the standard automated metric for text-to-speech (TTS), is in practice computed from ASR output that drops those marks: a synthesizer can ace CER and still say vehicle for husband. We introduce DunDun -- named for the dùndún, the Yorùbá talking drum that speaks through pitch alone -- an automated, reference-free lexical-tone metric that needs no tone-labelled corpus. The gold High/Mid/Low sequence is read from the input text's diacritics (in TTS that text exists by construction, so no reference recording is needed); the prediction comes from the audio's pitch track. We validate three ways. Flattening pitch with PSOLA resynthesis collapses DunDun while CER does not move. Inverting High and Low in the answer key of 300 native recordings drives the two-class readout to 0.14, symmetrically below its 0.35 chance level -- a consistency check on the scoring path, not independent evidence. And three native listeners, over 67 blind A/B trials, pick the tone-correct clip 89.6% of the time (95% CI 80.0-94.8; p < 1e-4); whether DunDun tracks those judgements trial by trial is not resolved at this sample size. Applied to a massively multilingual zero-shot TTS model, DunDun shows what CER cannot: Yorùbá tone sits near the native anchor before any Yorùbá fine-tuning (0.567 +/- 0.02 over five decode seeds vs. 0.596; chance 0.33), despite the 21.4% CER the model's own paper reports; and a few hours of clean audio halve CER (5.6% to 2.7% by 5h, 1.7% by 15h) while tone saturates within the hour. On non-tonal Swahili, CER already captures the gains: the metric a language needs is language-dependent. We release the metric and the complete validation protocol.