Voiceprint optimization improves real time streaming speech conversion
VOSSA: Voiceprint Optimization for Streaming Speech Architectures
Computation and LanguageMachine Learning
Summary
Real-time voice conversion systems often use speaker features made to ignore changes in speech style, which can limit sound quality when converting speech live. The authors developed VOSSA, a method that extracts speaker voice features from the vocal content itself and learns them together with the voice conversion task. This approach helps keep important voice details like pitch changes and vowel sounds, leading to more natural and clear voice conversions. Tests on several datasets show VOSSA matches or beats existing methods in quality and speaker similarity.
What this means in practice
- •For speech technology developers: Build streaming voice conversion tools that better preserve pitch and vowel details without separate speaker models.
- •For call center technology teams: Improve live voice transformation systems in customer service to enhance clarity and speaker resemblance.
Authors
Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah, Ricardo Gutierrez-Osuna
Abstract
Real-time voice conversion (VC) systems commonly rely on pretrained speaker embeddings from automatic speaker verification (ASV) models. While effective for speaker discrimination, these embeddings are trained to remain stable across phonetic and prosodic variations within-speaker, which may conflict with frame-level acoustic generation in streaming constraints. To address this issue, we propose VOSSA (Voiceprint Optimization for Streaming Speech Architectures), a speaker representation framework that extracts speaker information from intermediate content encoder layers and aggregates using attentive statistics pooling. The embedding is trained jointly with VC objectives, removing the need for a separate speaker encoder. Across six datasets, VOSSA improves F0 dynamics and vowel-discriminative acoustic cues while maintaining comparable NISQA-MOS, WER, and speaker similarity. Perceptual tests further indicate improvements in naturalness, speaker similarity, intelligibility, and vibrancy.