Cost efficient evaluator improves live streaming speech prosody judgments
Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis
SoundArtificial Intelligence
Summary
Evaluating how natural and expressive computer-generated speech sounds is tricky because it involves many details like emotion and energy. The authors created a cheaper tool called Live-ProsodyJudge (LPJ) that mimics a costly expert system to judge speech quality. They also fixed a problem where the tool would mix up different speech qualities into one overall score, making judgments less detailed. Their improved version, D-LPJ, can tell apart different speech qualities and closely matches human opinions in tests.
What this means in practice
- •For speech technology teams: Select the best expressive speech output from multiple TTS options using a cost-effective prosody judge closely matching human preferences.
- •For voice assistant developers: Improve the naturalness of real-time TTS voices by incorporating fine-grained multi-dimensional prosody feedback during model training.
Authors
Zifan Guan, Longyu Lu, Junan Zhang, Zhizheng Wu, Meiguang Jin, Junfeng Ma
Abstract
Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.