CycleSpeech improves speech synthesis with two-way voice profile feedback
CycleSpeech: Reciprocal Alignment for Instruction-Controlled Speech Synthesis and Paralinguistic Understanding
Sound
Summary
Speech synthesis systems often create voice outputs based on instructions, but they usually don't learn from how well those voices are understood. The authors created CycleSpeech, which links speech generation and understanding through a shared voice profile that both guides and checks the speech. This two-way process helps the system better match instructions and capture speaking styles without needing extra human feedback. Tests in Chinese and English show that CycleSpeech improves how closely the generated speech follows instructions while keeping voice quality high.
What this means in practice
- •For voice assistant developers: Improve voice assistants by generating speech that better follows user instructions about speaking style or emotion without extra human-labeled preferences.
- •For call center technology teams: Enhance automated agents to produce speech that matches specific voice profiles, enabling more consistent and natural interactions with customers.
Authors
Huan Liao, Haonan Han, Xingwen Han, Dekun Chen, Yuancheng Wang, Zhizheng Wu
Abstract
Instruction-controlled speech synthesis and paralinguistic understanding are often trained independently, leaving reciprocal feedback between the two tasks underexplored. We introduce CycleSpeech, a framework that connects generation and understanding through a shared, structured voice profile that serves as a common target for supervision and reciprocal feedback. The forward cycle assesses whether synthesized speech expresses the intended attributes by comparing recovered and target profiles. The backward cycle evaluates whether profiles inferred from real speech can guide reconstruction of the source speaking style. To support both directions, we construct a bilingual dataset of 20,046 examples pairing instructions, target speech, speaker references, and structured profiles. Building on joint supervised fine-tuning, CycleGRPO alternates policy updates using reciprocal rewards grounded in profile consistency and speaking-style reconstruction. Fixed target profiles anchor feedback from the evolving counterpart. This procedure requires neither human preference annotations nor an additional preference-trained reward model. Evaluations on Chinese and English benchmarks show improved instruction adherence and profile recovery while maintaining competitive synthesis quality. Compared with Step-Audio-2-mini, CycleSpeech improves instruction-match accuracy by 4.50 and 10.06 percentage points in Chinese and English, respectively. Controlled ablations further support the contribution of cycle feedback to generation control. These results support structured voice profiles as an interface for reciprocal training between speech generation and paralinguistic understanding. An online demo is available at https://cyclespeech.github.io.