Compact speech synthesis model balances detail and speed efficiently
GrainSpeech: Less Context, More Detail for Compact Speech Synthesis
Artificial IntelligenceSound
Summary
Making small speech synthesis models that sound good is hard because they have to balance how much detail they capture with how big they are. The authors found that looking at a limited range of about 15 sounds (phonemes) is enough for predicting key speech features like pitch and duration. They created a small model with just 264.8 thousand parameters that runs very fast on tiny devices like microcontrollers, while sounding almost as good as much bigger models. They also improved how the model learns fine sound details using a new method tailored for audio features.
What this means in practice
- •For embedded device developers: Create efficient speech generation systems that run in real time on resource-limited microcontrollers using a compact model with fewer parameters.
- •For mobile app developers: Integrate lightweight speech synthesis features that maintain quality while greatly reducing model size and computational demand.
Authors
Zitao Liang, Chang Gao
Abstract
Compact acoustic models face a challenging quality-capacity trade-off. We investigate two factors in this regime: encoder context and Mel-spectrogram supervision. A receptive-field-scaling study shows that expanding self-attention beyond 15 phonemes provides no consistent gains in pitch, energy, or duration prediction. Guided by this finding, we introduce a fixed-receptive-field convolutional encoder that reduces the respective prediction errors by 36.0%, 17.3%, and 3.4%. We further show that directly transferring image-domain gradient-variance supervision restores fine-scale variation but degrades predicted quality, motivating a Mel-specific formulation with axis-specific gradients, overlapping local statistics, and log-domain variance matching. GrainSpeech contains only 264.8K parameters and achieves 17.9x real-time Mel generation on a microcontroller (MCU), while attaining UTMOS scores comparable to substantially larger models with less than 1.5% of their parameters. Source code and demos are available at https://github.com/lab-emi/GrainSpeech.