Unified model improves speech recognition for target speakers with text and voice cues
Unified Target-Speaker ASR with Text and Enrollment Speech Cues
Sound
Summary
Recognizing what one specific person says in a noisy room with many speakers is hard. The paper presents a new method that uses two types of clues together: a short hint of the spoken words and a voice sample of the target speaker. By combining these hints in a single system, the method more accurately understands the target speaker’s words, even with other voices interfering. The authors show their approach works better than using either clue alone or combining them separately.
What this means in practice
- •For voice assistant developers: Build voice assistants that better recognize a designated user’s commands among other voices using combined text and voice sample cues.$Commercial implications: Enables improved commercial voice assistants to accurately isolate a user’s speech in multi-speaker settings, enhancing personalized device control.
- •For call center software engineers: Integrate dual-cue speech recognition systems to identify a target speaker more reliably when multiple agents or customers speak simultaneously.
Authors
Yuxiang Mei, Yuchen Yan, Dongxing Xu, Jiaen Liang, Yanhua Long
Abstract
Target-speaker automatic speech recognition (TS-ASR) aims to recognize a designated speaker while suppressing interfering speech in multi-talker environments. Conventional TS-ASR typically relies on an enrollment utterance, whereas text-guided methods use known lexical content, such as a wake word, to identify the target speaker from the observed mixture. These two cues provide complementary information but are usually studied separately. We propose a Unified Dual-Cue TS-ASR framework that supports text cues, enrollment speech, or both within a single model. Text cues interact with the mixture representation to extract target-speaker information conditioned on known lexical content, while an independent enrollment utterance provides complementary speaker information. Cross-attention cue-conditioning modules are integrated into shared Conformer blocks, and negative-cue sampling provides cue-validity supervision during dual-cue training. Experiments on 30,000 two-speaker mixtures across five recording/domain conditions and four oracle text-cue lengths show that, with five-character text cues, the concatenated dual-cue method achieves 8.80% CER, compared with 17.32% for text-only and 29.06% for enrollment-only inference. It also outperforms parallel dual-cue fusion (9.49% CER) and yields lower dual-cue CER across all five evaluation subsets. These results demonstrate the benefit of jointly exploiting complementary lexical and speaker information for target-speaker ASR.