Speaker anchored routing improves full duplex dialogue audio recovery
Beyond Two Tracks: Speaker-Anchored Full-Duplex Dialogue Recovery
SoundComputation and Language
Summary
Conversations usually happen one person at a time, but as technology moves toward hearing both people speak at once, it becomes tricky to keep their voices separate and correctly attributed. The authors developed a way to link separated audio streams back to the right speaker consistently through a process called speaker-anchored routing. This reduces errors where voices get mixed up over time, making recovered conversations clearer and more accurate without changing the audio quality. Their approach works well even on new types of recordings that the system wasn't trained on.
What this means in practice
- •For speech recognition teams: Improve speaker tracking in multi-speaker audio to enhance transcription accuracy in full-duplex conversational systems.
- •For call center platform developers: Maintain consistent speaker identity tracking to enable real-time conversation analysis and improved agent performance feedback.$Commercial implications: Enables development of commercial tools that reliably separate and attribute overlapping speech in customer service calls for better analytics.
Authors
Yuxuan Chen, Yang Xiao, Jack Sun, Chak Sun, Brick Li, Sam
Abstract
Spoken interaction is moving from alternating turns to full-duplex conversation, where an agent listens while speaking. Learning this coordination requires natural dialogue with persistent speaker-resolved tracks, a data requirement that remains difficult to satisfy at scale. Generative recovery offers a route from ordinary recordings, but recovering two voices is not enough: their track assignments can drift as the conversation unfolds. We address this missing link with speaker-anchored routing before diffusion: a 1.84M-parameter module matches the separator's coarse streams to fixed enrollment identities and orders them before a frozen generator reconstructs the speech, without retraining the generator or altering the reconstructed audio. On 25-minute remote conversations, routing cuts wrong-track time from 43.9% to 4.1%, below the 6.2% of matched-enrollment waveform re-identification, with its largest advantage when one participant holds the floor (17.8% versus 28.6%), and raises fixed-track activity F1 from 0.46 to 0.84 while per-chunk reconstruction quality is unchanged. The same router transfers to unseen face-to-face and telephone recordings, and a margin-gated verification step brings their wrong-track time to 0.9-1.7% while embedding 20-36% of the output, where full re-identification embeds all of it. Recovered voices thus become the participant-consistent tracks that full-duplex dialogue modeling needs. Audio examples: https://elpsykongloo.github.io/beyond-two-tracks/