Speech recognition improves by asking users to repeat unclear parts
Ask, Don't Guess: Targeted Clarification for Speech Recognition Errors
Computation and Language
Summary
Speech recognition often makes mistakes when people talk from a distance or in noisy places. The authors propose a system that figures out which parts of the recognized speech might be wrong and asks the speaker to repeat only those parts more clearly. This targeted asking helps fix errors without bothering the speaker too much. Their method uses both noisy and clean recordings to learn when repeating will help, and it uses language models to revise only the suspected mistakes. Overall, their approach catches more errors and reduces mistakes by asking for fewer repeated words.
What this means in practice
- •For voice assistant developers: Improve far-field speech recognition by selectively prompting users to repeat unclear phrases, increasing transcription accuracy without frequent interruptions.$Commercial implications: Enables more reliable voice assistants by reducing transcription errors through targeted user interaction, enhancing user experience and system performance.
- •For call center technology teams: Deploy targeted clarification prompts in real-time speech systems to reduce transcription mistakes and improve customer-agent interaction quality.
Authors
Sankalp Parashar, Preethi Jyothi, Sunita Sarawagi
Abstract
Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during streaming ASR permits a recourse mechanism where the speaker can be asked to repeat the affected content under cleaner acoustic conditions. We propose \sysname, a new framework for online error localization, that identifies spans likely to benefit from this intervention. Our supervision uses clean audio to estimate how the recognizer's output might change under improved acoustic conditions. We probabilistically align noisy and clean decodes using a pair-HMM to obtain soft supervision. These soft targets train a counterfactual gain scorer by combining a suite of acoustic and language features to produce token-level scores, that are further grouped into spans for targeted recourse. An LLM uses the ASR transcript of the repetition to revise only the selected span. At a precision of $70\%$, \sysname achieves the highest recall by identifying up to 52\% more erroneous tokens compared to the strongest baseline. Targeted repetition yields relative WER reductions of up to 24\% under a budget that accounts for both interruption frequency and the lengths of re-uttered spans.