End to end speech recognition excludes opt out speakers dynamically
Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition
Computation and LanguageSound
Summary
Some people may not want their voice transcribed during meetings. The authors present a way for speech recognition systems to skip transcribing voices of people who choose to opt out, while still showing when those speakers talk. They developed a small add-on module that works with existing speech models to do this on the fly, even for new opt-out speakers not seen before. Their tests with English and Mandarin meeting recordings showed the system could reduce transcription accuracy for opt-out speakers significantly without affecting other speakers. This could help protect privacy in large-scale online meetings.
What this means in practice
- •For video conferencing platform developers: Integrate dynamic speaker opt-out features to protect user speech privacy during live transcriptions in multi-participant meetings.$Commercial implications: Enables privacy-compliant transcription services suitable for commercial conferencing products by allowing selective speech exclusion.
- •For speech recognition engineers: Improve multi-speaker ASR systems by adding lightweight modules to exclude specified speakers dynamically without retraining entire models.
Authors
Bo Su, Yueru Yan, Thai Le
Abstract
We introduce target-speaker unlearning ASR (TSU-ASR) task in a fully end-to-end framework for multi-speaker ASR and diarization. Given a multi-speaker utterance and a set of opt-out speakers who do not wish to have their speech transcribed, the task requires an ASR system to transcribe all speakers except the opt-out ones, while still indicating when those speakers are active. As a first step towards tackling this task, we introduce a novel, light-weight Enrollment-Conditioned Gating (ECG) module attachable to a frozen dual-stream speech LLM that enables ASR for new opt-out speakers dynamically during inference, even those who were not seen during initial ECG training phase. Our experiments on both AMI (English) and AliMeeting (Mandarin) datasets show that speech transcription accuracy for corresponding opt-out words or characters falls from 72.3% to 48.2% and from 73.6% to 27.3%, respectively, while retained speakers' transcription error rates maintain more or less the same. Our approach provides a practical solution for modern video conferencing platforms, allowing speakers to dynamically opt-out from automated AI transcriptions without forcefully leaving the meeting sessions, enabling a privacy-preserving interface for potentially millions of online meetings daily.