Conversational agents adjust speech in noise for better clarity
Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization
Sound
Summary
Background noise makes it hard for voice assistants to talk clearly. The authors created a system that mimics how people naturally speak louder and clearer in noisy places, called the Lombard effect. Their method teaches computers to change their voice based on noise levels, making speech easier to understand in different sounds environments. Tests show this system improves clarity as noise gets louder.
What this means in practice
- •For voice assistant developers: Enhance voice assistants to automatically adjust speech clarity in noisy environments, improving user communication without manual tuning.
- •For call center software teams: Integrate noise-aware speech synthesis to maintain clear computer-generated responses amid background noise in customer support calls.
Authors
Zhaoyang Wang, Dingding Yao, Chengzhong Wang, Junfeng Li
Abstract
Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.