Speaker diarization improves by combining neural and embedding methods

Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization

SoundMachine Learning

Summary

Identifying when different people speak in audio recordings can be done by using methods that focus on voice embeddings or neural network-based approaches. These two methods work well but use different kinds of information that don't easily fit together. The authors propose a new way to combine these methods without needing to retrain anything, making the process more effective. Their approach mixes the strengths of both methods and tested on real-world datasets showed it reduces errors in distinguishing speakers and improves the clarity of transcripts.

What this means in practice

  • For voice assistant developers: Enhance speaker separation in multi-user voice interactions by integrating neural and embedding-based diarization without extra training.$Commercial implications: Enables improved multi-speaker recognition products by combining methods in a single process without costly retraining.
  • For call center analytics teams: Improve the accuracy of speaker identification in recorded customer calls by applying a fused diarization approach that reduces error rates.

Authors

Yehoshua Dissen, Joseph Keshet, Eduard Golshtein

Abstract

Speaker diarization systems based on speaker embeddings and neural diarization exploit complementary forms of speaker information, but their intermediate representations are not directly compatible. We introduce Training-Free Affinity Fusion (TFAF), which integrates the speaker structure inferred by a neural diarizer into an embedding-based diarization system. The neural speaker partition is used to condition local speaker representations, from which we construct a continuous affinity matrix and combine it with the embedding-based acoustic affinity before a single global clustering step. The method requires no additional training, shared embedding space, speaker-label alignment, or hard transfer of the neural diarizer's speaker count. Experiments on AMI and CALLHOME show consistent DER improvements over both constituent systems; on AMI, fusion also improves speaker-attributed transcription. Ablations show that the neural speaker partition accounts for most of the gain, while retaining the continuous embedding-based affinities provides additional benefit over hard partition fusion.