Papers for

virtual role-play designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Speech models improve natural conversation by adding human sounds

Speech LLMs That Breathe: Modeling Non-Verbal Vocalizations in Dialogue

Abstract: The user experience of speech interaction models has improved, benefiting from the modeling of paralinguistic attributes and advances in speech reconstruction paradigms. However, human communication follows extensive norms regarding non-verbal vocalizations (NVVs) spanning respiratory, affective, and discourse. The omission of these events imposes inherent limitations on the naturalness of current approaches. In this work, we cover 13 major types of NVVs and construct a natural dialogue dataset that conforms to human conversational norms through an automated synthesis pipeline. We enhance the ability of speech interaction models to generate contextually appropriate NVV events through multi-stage fine-tuning and NVV-aware GRPO (Group Relative Policy Optimization). Based on distinct modeling paradigms, we substantially improve the expressiveness of VocalNet and VITA-Qinyu in empathy interactions and role-play dialogues, and further discuss the architectural necessity for achieving high-quality natural speech generation.

Mon 19 OctComputation and Language

The gist

Current speech interaction models often miss important human sounds, like sighs, breaths, or laughter, which limits how natural they feel. The authors studied 13 key types of these non-verbal vocalizations and built a new dataset that follows human conversational rules. They then taught speech models to generate these sounds appropriately in different contexts, improving how expressive and empathetic dialogues sound. This work highlights the need to include these sounds for more lifelike speech systems.

Open → 2610.25942v1