Preference optimization improves control of nonverbal vocalizations in speech models
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Sound
Summary
Generating natural nonverbal sounds, like laughs or sighs, that fit speech is challenging because it's hard to know which variations sound better. The authors created a method that uses a large audio-language AI to compare and rank these sounds without needing people to label them. They then teach the model to prefer better sounds using this AI feedback, making the generated nonverbal vocalizations more accurate and natural. Their approach showed better results on a public test compared to previous methods.
What this means in practice
- •For speech synthesis developers: Create speech systems that generate more natural and context-appropriate nonverbal sounds without needing human preference labels.
- •For voice assistant engineers: Improve expressiveness of voice assistants by adding controllable nonverbal vocalizations that better match user interaction contexts.
Authors
Jingbin Hu, Qirui Zhan, Yuang Cao, Ziyu Zhang, Yunxiang Chen, Houdun Liu, Shuo Feng, Bengu Wu, Lei Xie, Liumeng Xue
Abstract
We propose a preference optimization framework with Large Audio-Language Model (LALM) feedback for controllable non-verbal vocalization (NVV) generation in continuous autoregressive speech models. To construct preference data without human preference annotation, we build a bilingual prompt corpus by combining NVV-injected real transcripts with LLM-generated semantically aligned prompts, perform stochastic model rollouts, and use a LALM to rank candidate utterances and form same-prompt chosen--rejected pairs. We then adopt a two-stage optimization strategy: Rejection Sampling Fine-Tuning (RSFT) first adapts the model to LALM-selected high-scoring samples, followed by Anchored Flow-DPO, which formulates pairwise preference optimization using utterance-level flow-matching loss and retains the chosen-sample flow-matching objective as an SFT anchor. This design enables DPO-style preference learning without explicit sequence likelihoods while preserving direct supervision on preferred realizations. On the official 1,600-utterance NVVSpeech Challenge Track~2 test set, our method achieves a Final Track2Score of \textbf{75.80} (79.39 ZH / 72.21 EN), outperforming the VoxCPM2 baseline by \textbf{+1.84}. The improvements are mainly driven by higher NVV Accuracy and NVV Perceptual Effect, while Overall Quality remains stable.