Image models reused to create clearer and faster speech synthesis
ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis
SoundMultimedia
Summary
Making computers produce speech that sounds like a particular person is tricky and needs a lot of training. This paper shows it is possible to use parts of artificial intelligence models trained on pictures to help make speech. The authors keep some image-processing parts frozen and only retrain certain components to make spoken words that are clearer and sound like the original speaker. They show this approach works better and faster than starting from scratch and can even follow voice instructions or use different speaking styles.
What this means in practice
- •For voice assistant developers: Create speech synthesis systems that sound clearer and preserve speaker identity by adapting pretrained image diffusion models.$Commercial implications: Enables development of high-quality, customizable TTS products with less training time for consumer voice assistants and smart devices.
- •For speech technology engineers: Speed up training and improve accuracy of text-to-speech systems by initializing with pretrained image diffusion transformers.
Authors
Hanchen Pei, Shujie Liu, Yayue Deng, Fangyun Wei, Yanqing Liu, Jianwei Yu, Gongping Huang
Abstract
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.