MeanVoiceFlow2 speeds up zero-shot voice conversion with quality
MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion
SoundArtificial IntelligenceMachine Learning
Summary
Voice conversion changes one person's speech to sound like another's. Existing good methods are slow because they use complex steps to understand the speech content. The authors developed MeanVoiceFlow2, a new approach that trains the speech conversion and content understanding parts together to make the process about nine times faster. It keeps the quality and speaker likeness high while speeding things up. This could help use voice conversion more easily in real-time or mobile settings.
What this means in practice
- •For speech technology developers: Create faster zero-shot voice conversion tools that maintain high quality for real-time voice applications.
- •For game audio engineers: Integrate rapid voice conversion to modify character voices dynamically without sacrificing audio quality.
Authors
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, Yuto Kondo
Abstract
Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately $9\times$ faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.