Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
SoundArtificial Intelligence
Summary
The authors created LoopTTS, a system that improves speech generated by text-to-speech (TTS) models by fixing small but noticeable problems like wrong emphasis or awkward pauses. Their method uses an audio-based AI judge to find these issues and then gives detailed instructions to a special TTS refiner model, which adjusts the speech accordingly. They trained this refiner with a large dataset labeled for prosody (the rhythm and intonation of speech). Human tests showed LoopTTS fixes errors better than just regenerating the speech or using the original model. Overall, the system helps make TTS sound more natural by targeting prosody mistakes specifically.
Authors
Zeyang Song, Tianchi Liu, Tianrui Wang, Chenglin Xu, Steven Y. Guo, Haizhou Li
Abstract
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.