NVAlign improves control of non-verbal sounds in speech synthesis
NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech
SoundArtificial IntelligenceComputation and Language
Summary
Text-to-speech (TTS) systems can now add non-verbal sounds like sighs or laughs during speech, but making these sounds happen accurately is tough. The authors developed NVAlign, a method that fine-tunes TTS models after training using feedback from a specialized speech recognition model that understands these non-verbal sounds. This approach helps the TTS system follow instructions for when to add such sounds more precisely, while keeping the speaker’s voice natural and clear. Tests and listener feedback show NVAlign works better than previous methods at controlling non-verbal sounds in generated speech.
What this means in practice
- •For speech synthesis engineers: Enhance control over non-verbal sounds in TTS systems to improve expressiveness and naturalness in generated speech.
- •For voice assistant developers: Improve voice assistants’ ability to include non-verbal sounds for more engaging and human-like interactions.
Authors
Qiaolin Wang, Pedro Sandoval-Segura, Anunaya Joshi, Edvardas Jurkonis, Jake Downie
Abstract
While modern text-to-speech (TTS) systems generate highly natural speech and support inline non-verbal vocalization (NVV) tags, accurate control over these events remains challenging. A key gap is the lack of established post-training methods for non-verbal control in continuous autoregressive flow-matching TTS. To this end, we present NVAlign, a direct-gradient post-training framework for NVV tag-following in this architecture. We first perform supervised fine-tuning (SFT) of TTS models and an NVV-aware automatic speech recognition (NV-ASR) model on NVV-annotated speech, then freeze the NV-ASR model to serve as the reward model for post-training. A two-step gradient surrogate enables efficient reward backpropagation through the flow-matching sampler to jointly update the autoregressive backbone and acoustic flow head. Fidelity penalties and reference-velocity regularization help preserve speaker similarity and speech quality. Results from NVV-SuperBench and human listening evaluations show that NVAlign improves tag-following accuracy over SFT and Flow-GRPO baselines. These findings demonstrate that direct reward-gradient optimization can improve non-verbal control in continuous autoregressive flow-matching TTS. Audio samples are available at https://nvalign.github.io/.