Speaker-adapted speech system improves expressive control in long audio streams
SCIC: Scope- and Codebook-Aware Instruction Conditioning for Speaker-Adapted Expressive TTS
SoundArtificial Intelligence
Summary
When computers read long texts aloud, it can be hard to make their voice sound natural and expressive throughout. The authors found a way to better control the way the computer changes pitch, energy, and speed at the level of each sentence part, based on the speaker’s previous speech. They built a system called SCIC that uses special codes to link spoken instructions with these speech changes, improving control without hurting speech quality or how well the system sounds like the speaker. Their system creates more natural-sounding paragraphs with clearer expression differences.
What this means in practice
- •For voice assistant developers: Build voice assistants that maintain natural and varied expressiveness over long spoken responses using speaker-relative prosody controls.
- •For audiobook producers: Generate long-form narration with improved paragraph-level expression and speaker style consistency for more engaging audiobooks.$Commercial implications: Enables scalable production of expressive audiobooks with consistent speaker voice and nuanced prosody control for publishers.
Authors
Longyu Lu, Zongwei Du, Mengtao Xing, Zhuoqun Liu, Zifan Guan, Meiguang Jin, Junfeng Ma
Abstract
Long-form live-streaming TTS requires context-dependent prosody and paragraph-level coherence. However, many existing instruction-based TTS systems use global or uniform conditions, providing limited explicit control over clause-level relative prosodic changes. We introduce Speaker-Relative Inline Prosody Control, where each Pitch, Energy, or Speed instruction targets a clause relative to the preceding clause from the same speaker, while Pause uses an absolute duration interval. In codec-based TTS, Speed and Pause affect sequence length, whereas Pitch and Energy rely on residual codebooks. By analyzing Qwen3-TTS RVQ codebooks, we find that Energy concentrates in early residual codebooks, whereas Pitch accumulates across a deeper prefix. We therefore propose Scope- and Codebook-Aware Instruction Conditioning (SCIC), combining a Temporal Instruction Router for frame-level tag activation with Tag-Specific Codebook Weighting over residual codebooks. SCIC improves speaker-relative Pitch and Energy control over standard instruction fine-tuning using text-token tags. We further apply multi-reward GDPO post-training to jointly optimize control and quality, improving control accuracy while preserving CER and speaker similarity. In long-form synthesis, SCIC produces a more distinct paragraph-level expressive hierarchy than speaker-adapted SFT without instructions. Audio demos are available at: https://taoliveaigc.github.io/SCIC/