Voice isolator reduces word errors in noisy phone calls
The Voice on the Line
SoundComputation and Language
Summary
Phone call microphones pick up all sounds nearby, making it hard to hear only one speaker. The authors trained smart voice isolators that can pick out a caller's voice even with other voices around, working in nearly real-time. They improved accuracy by enrolling the caller from the call itself and using special training methods on longer conversations. Their released model runs efficiently on a single CPU and improves word error rates compared to previous tools.
What this means in practice
- •For telecommunication platform developers: Integrate improved voice isolators into call platforms to isolate callers' voices more accurately in noisy environments with low latency.$Commercial implications: Enables building better commercial phone and video call services with clearer voice separation, improving user experience.
- •For meeting transcription services: Use the model to enhance transcription accuracy by isolating individual speakers in multi-person meetings, even with headset crosstalk.
Authors
Husein Zolkepli
Abstract
A transcriber listening through one microphone also hears everyone near it. We train causal voice isolators: masking transformers of 140M, 243M and 374M parameters, 20 ms of latency, enrolling the caller from the call itself. Short evaluation windows hide two failures that real calls expose: forgetting the caller when another voice talks alone for a while, and damaging a caller who is alone. Long calls with turns, memory beyond the attention window, real headset crosstalk and drifting caller levels address both, but each trades one against the other. WavLM-Large's own weights, made causal and trained with the model, move the trade: M with them (555M parameters) lowers the error on our development sets from 11.44% to 10.55%; a student trained from scratch to imitate WavLM does not, and w2v-BERT 2.0 and Whisper's encoder, pre-trained on far more audio, help less or not at all. A lone caller loses most at the start of a recording: no training call opened with someone already talking. Training on windows cut from longer calls lowers the error on clean development calls from 3.92% to 2.96%; this is our release. Live, the release makes 16.13% word errors on 60 s AMI windows, against 23.66% for VoxLock, the strongest open isolator, and 10.19% on its own AMI windows, against 11.71%; it is level on Krisp's benchmark (5.74% against 5.63%), and a caller who is alone still loses more words than with VoxLock (5.58% against 2.92% on clean test calls). Distilled into 22.1M parameters without WavLM, it runs live on one CPU thread inside a LiveKit agent, at 33% of a core per call in 20 ms frames, with 10.66% error on our development sets (the release 10.49%). We open-source the release and this CPU model, with a LiveKit plugin, at https://huggingface.co/collections/Scicom-intl/voice-isolator.