Preference post-training improves continuous autoregressive speech models
What Matters in Preference Post-Training of Continuous Autoregressive Speech Language Models
Sound
Summary
Making speech generation better is tricky because the models involve many parts that influence each other. The authors show that when retraining these models to prefer “better” speech, changing one part of the model’s structure can help more than just adjusting the final step. They also found that bigger changes inside the model don’t always mean the speech sounds better. Their work helps figure out how to train these speech models so they make more natural and higher-quality audio.
What this means in practice
- •For speech synthesis engineers: Improve speech quality by selectively retraining parts of continuous autoregressive speech models using preference-based feedback.
- •For voice assistant developers: Enhance naturalness of generated speech by applying preference post-training strategies to autoregressive speech models.
Authors
Yingda Shen, Yao Qian, Dekun Chen, Yufei Liu, Junan Zhang, Chong Jing, Han Wang, Yuxiang Wang, Zhizheng Wu
Abstract
Continuous autoregressive speech language models combine an autoregressive (AR) backbone with a diffusion or flow-matching head. Preference post-training requires choosing how to construct pairs and which modules to update. Controlled interventions show that a perturbed acoustic patch changes subsequent backbone states, yet larger representation changes do not necessarily reduce speech quality. We construct preference pairs through noise or inference-configuration perturbations and formulate a preference update based on reference-relative flow-matching errors. Updating the backbone with a fixed head yields stronger reward improvements than head-only adaptation in our evaluated setting. Across flow-time and module ablations, the lowest training loss does not consistently produce the best speech. These findings support preference post-training that considers both acoustic sampling and autoregressive conditioning, with training choices validated on generated utterances.