Training free method improves pronunciation transcription accuracy
Training-Free Pronunciation Transcription via Text-Constrained Acoustic Rescoring
Computation and LanguageSound
Summary
Transcribing how words are pronounced accurately and efficiently is important for creating text-to-speech systems. The authors found that existing methods either use only text, only speech, or both but require expensive data and training. They propose a new way to combine both text and speech information during use, without needing extra training. This approach improves accuracy and speed on multiple languages and beats previous methods, including commercial ones.
What this means in practice
- •For text-to-speech engineers: Generate highly accurate pronunciation transcriptions by combining text candidates and acoustic scoring without additional model training.
- •For speech recognition developers: Improve phonetic transcription accuracy by rescoring pronunciation candidates with pretrained speech-to-pronunciation models, even using imperfect automatic transcripts.
Authors
Hikaru Asano, Yotaro Kubo, So Kuroki
Abstract
Accurate and efficient pronunciation transcription is essential for preparing text-to-speech training data at scale. Existing approaches have different limitations: grapheme-to-pronunciation (G2P) and speech-to-pronunciation (S2P) methods each capture only partial information, using only text or only speech, while speech-and-text-to-pronunciation (ST2P) methods use both but require costly pronunciation-annotated data. To address this problem, we propose a training-free ST2P pipeline that integrates both lexical and acoustic information at inference time. Lexical resources and G2P tools generate text-constrained candidates, and a left-to-right greedy search selects the best one using whole-sequence negative log-likelihoods from frozen pretrained S2P models. On three Japanese corpora, our method reduces Character Error Rate (CER) from 0.60--1.40\% (text-only baseline) to 0.04--0.17\% with reference transcripts, and 0.64--1.58\% with ASR transcripts. It outperforms all baselines, including a trained ST2P model and commercial multimodal LLMs. Our greedy search method is 3--3.5$\times$ faster than beam search at similar CER, and the cascade is 2$\times$ faster than direct decoding ensuring the efficiency and accuracy. In Spanish, French, and preliminary English, it also surpasses four open multimodal LLMs and the best traditional methods.