Target speaker speech recognition improves by tuning guidance scales
Asymmetric Classifier-Free Guidance for Target-Speaker ASR
Computation and LanguageSound
Summary
Recognizing what a specific person says in noisy or overlapping conversations is hard. The authors show that adjusting how much the recognition system focuses on the target speaker during transcription can help. They combine predictions that include and exclude speaker information and change their influence based on conditions. Their approach reduces errors by picking the right balance for each recording, even when conditions change.
What this means in practice
- •For call center operations: Improve transcription accuracy for a specific speaker in noisy multi-person calls by adjusting model guidance per conversation.
- •For conference transcription teams: Deliver clearer transcriptions isolating individual speakers in meetings with overlapping speech or background noise.
Authors
Yiwen Guan, Jacob Whitehill
Abstract
Target-speaker automatic speech recognition (TS-ASR) must identify and transcribe a desired speaker under varying overlap and noise conditions. These changes alter the acoustic evidence for the target speaker in the speech mixture, motivating inference-time calibration of speaker conditioning. We introduce asymmetric classifier-free guidance (CFG) for TS-ASR using Whisper: the speaker-conditioned branch predicts the target transcript, while the speaker-unconditioned branch predicts serialized multi-speaker transcripts. CFG adjusts the contribution of speaker conditioning during decoding through a single guidance scale. We select a global guidance scale on target-domain development data and train a lightweight encoder-based predictor to adjust it for each utterance, keeping the recognition model fixed. Under domain shifts, our full system achieves relative word error rate (WER) reductions of up to 21.8% over the condition-only baseline, and 5.6% over standard conditional decoding of the same CFG-trained model. Oracle analysis shows that substantially larger WER reductions are possible through utterance-level scale selection and identifies how beneficial adjustments vary with domain shifts.