Neural tracker uses speaker identity to improve speech source tracking
Identity-Assisted Association of Unordered DOA Estimates for Neural Speech Source Tracking
Sound
Summary
Tracking where multiple people are speaking is difficult because voices can overlap and move around. The authors created a new method that keeps track of who is speaking by combining where the sound comes from with unique voice features. They built a neural network that learns to keep sounds linked to the right speakers over time. This approach helps reduce confusion when sounds are close together or happen one after another.
What this means in practice
- •For audio engineers: Improve multi-speaker localization in noisy environments by integrating speaker identity with spatial cues for stable tracking.
- •For conference system developers: Develop smarter microphone arrays that maintain consistent speaker tracking even with overlapping or intermittent speech.
Authors
Bing Yang, Di Liang, Xiaofei Li
Abstract
Tracking speech sources remains a challenge due to ambiguous data association arising from intermittent speech, close spatial proximity, and complex acoustic conditions. To address these issues, we propose an identity-assisted association that maps unordered direction-of-arrival (DOA) estimates to speaker-consistent source trajectories for reliable speech source tracking. Specifically, speaker identity embeddings are directly integrated into the model input as a complementary cue to spatial features. This enables maintaining identity consistency by combining long-term time-invariant vocal identity characteristics with the short-term continuity of spatial cues. To effectively process these heterogeneous inputs while accommodating their distinct characteristics, we design a unified neural tracker. Within this model, time self-attention modules capture the temporal evolution of each source, while source self-attention modules distinguish between competing source tracks. Experimental results demonstrate the superiority of the proposed neural tracker in mitigating association confusion for speech source tracking.