Spectral embedding method cuts training data for expanding text to speech vocabularies
SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS
Machine Learning
Summary
Text-to-speech systems need lots of audio recordings to learn how to say new words, which is hard when recordings are scarce or for new languages. The authors present SEAM, a way to create word representations for unheard words by combining smaller sound parts while keeping their order. This method uses math techniques to align these new word forms to what the system already knows, letting models learn with much less audio. Their tests show SEAM matches or nearly matches standard methods but needs up to 75% less data.
What this means in practice
- •For speech synthesis developers: Expand TTS model vocabulary efficiently using fewer paired audio samples for low-resource or new languages.
- •For multilingual voice assistant teams: Add new words from different languages to TTS systems with minimal retraining, speeding up deployment in diverse markets.$Commercial implications: Enables commercial voice products to support low-resource languages with less data, reducing costs and time to market.
Authors
Walaa Othman, Baher Mohammad, Stamatios Lefkimmiatis, Ammar Ali
Abstract
Expanding the vocabulary of conditional text-to-speech (TTS) models typically requires extensive fine-tuning with paired audio, creating a major bottleneck for low-resource and cross-lingual adaptation. To address this, we introduce SEAM, a zero-audio initialization method that constructs embeddings for unseen tokens from known subtokens. Unlike standard pooling strategies that ignore sequential structure, SEAM applies a spectral filter to preserve temporal phonetic order and uses closed-form ridge regression to align the new embeddings with the pretrained conditioning manifold. Across three languages and two TTS architectures, SEAM approaches or matches baseline performance using up to four times less training data, reducing sample complexity by 75\% compared to random initialization and mean pooling. Our approach provides a lightweight, reproducible, and theoretically grounded solution for vocabulary expansion in low-resource speech synthesis.