Multi-granularity alignment speeds up target speaker extraction training
MG-REPA: Multi-Granularity Representation Alignment for Generative Target Speaker Extraction
Artificial Intelligence
Summary
Separating a target speaker’s voice from overlapping sounds can be slow to learn for AI models. The authors introduce a method that teaches the AI to recognize both overall speaker traits and detailed speech features by aligning with two different pre-trained models. This approach, called MG-REPA, helps the system learn faster and perform as well as long-trained alternatives. It mixes broad sound identity and moment-by-moment details to improve voice separation.
What this means in practice
- •For audio engineers: Build faster training systems for isolating a target speaker’s voice from mixed audio using multi-level alignment techniques.
- •For speech recognition developers: Improve source separation modules to enhance speech recognition accuracy in noisy environments by better capturing speaker identity and speech content.
Tested on one dataset.
Authors
Dohwan Kim, Jung-Woo Choi
Abstract
Generative target speaker extraction (TSE) models achieve high-quality extraction, but suffer from considerably slow training convergence. Representation Alignment (REPA) accelerates diffusion transformer training by aligning hidden states with a pre-trained Self-Supervised Learning (SSL) teacher. However, existing alignment in the speech and audio domain operates at a single granularity. For TSE, frame-level alignment leaves the global, time-invariant characteristics of the target speech implicit, whereas utterance-level alignment leaves the time-varying content implicit, even though extracting the target requires both. We propose Multi-Granularity Representation Alignment (MG-REPA), aligning the hidden states with an SSL model at the frame level and a speaker verification model at the utterance level. We further apply temporal normalization to the frame-level target, removing its utterance-level statistics so that each branch concentrates on its granularity. On Libri2Mix, MG-REPA reaches the OVRL and PESQ of a MeanFlow-TSE baseline with $3.0\times$ and $2.1\times$ fewer epochs, respectively. Trained for 750 epochs, it is competitive with the published 2,000-epoch baseline on intrusive metrics while improving non-intrusive metrics. An ablation demonstrates that MG-REPA outperforms alignment at a single granularity.