Speech models improve natural conversation by adding human sounds
Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue
Computation and Language
Summary
Current speech interaction models often miss important human sounds, like sighs, breaths, or laughter, which limits how natural they feel. The authors studied 13 key types of these non-verbal vocalizations and built a new dataset that follows human conversational rules. They then taught speech models to generate these sounds appropriately in different contexts, improving how expressive and empathetic dialogues sound. This work highlights the need to include these sounds for more lifelike speech systems.
What this means in practice
- •For voice assistant developers: Add natural human vocal sounds like sighs and breaths to improve user empathy in voice assistants.
- •For virtual role-play designers: Create more believable characters by generating context-aware non-verbal vocal sounds in scripted dialogues.
Authors
Heyang Liu, Lixin Zhang, Jiayi Huang, Ziyang Cheng, Yuhao Wang, Ronghua Wu, Qunshan Gu, Yanfeng Wang, Yu Wang
Abstract
The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.