Fastdiar speeds up streaming speaker identification on cpu
FASTDIAR: Frame-level speaker encoder for Streaming Diarization
Machine LearningSound
Summary
Speaker diarization means figuring out who spoke when in a conversation. The authors found that current methods waste computing power by analyzing speech in short chunks and overlap. They changed a popular speaker recognition model to work continuously, creating quick updates every 80 milliseconds from recent audio. Their system works accurately with little delay, especially when many people talk, and runs much faster than real time on a normal processor.
What this means in practice
- •For voice assistant developers: Build voice assistants that identify speakers in live conversations instantly on CPU-limited devices.$Commercial implications: Enables cost-effective, real-time speaker separation features in consumer devices without specialized hardware.
- •For call center technology teams: Implement fast speaker diarization in customer support calls to monitor who speaks when with minimal latency.
Authors
Nikita Torgashov, Okan Köpüklü
Abstract
Real-time conversational agents require speaker diarization that streams and runs on a CPU. Most systems apply an utterance-level speaker encoder to short, heavily overlapping chunks, which wastes computation and leaves the model optimized for the wrong task. We instead turn a state-of-the-art speaker recognition architecture into a causal frame-level encoder that reads the stream once and emits one embedding every 80~ms from a bounded two-second window of past audio, and pair it with online clustering that gates every update on the self-similarity of the stream. Trained only by distillation from an utterance-level teacher on simulated and out-of-domain mixtures, and evaluated with one fixed set of hyperparameters, the system is the most accurate streaming diarizer on low-overlap benchmarks at sub-second latency, degrades far less than cache-based systems as the number of speakers grows, and runs five times faster than real time on a single CPU thread.