AI summaryⓘ
The authors looked at how using learned rewards for training text-to-speech models affects how well the speech matches human preferences like style and naturalness. They found that each reward mainly improves the specific feature it targets, and different rewards can't just replace each other. To keep the speech accurate, they used a character error rate constraint during training. They also showed that a method called Best-of-8 reranking works about as well as their reinforcement learning approach, suggesting the learning helps package good speech behavior rather than always doing better than simpler methods. Overall, the authors recommend carefully choosing rewards based on how they relate to human listeners when improving speech models after initial training.
text-to-speechreinforcement learningGroup Relative Policy Optimizationperceptual rewardscharacter error rateBest-of-N rerankingsubjective predictorsspeech post-traininglistener preferences
Abstract
Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.