Acoustic judge reduces false insertions in speech recognition models
Grounded in Sound: Reinforcement Learning with a Frozen Acoustic Judge to Curb ASR Insertion Hallucinations
SoundMultimedia
Summary
Speech recognition systems sometimes add words that aren't actually spoken, especially when the audio is unclear. The authors improve these systems by using a special audio-based judge that stays fixed during training to check if the predicted words really match the sound. Their method cuts down these false insertions significantly without making other errors worse. This approach helps the system rely more on the actual sound rather than just guessing from common language patterns.
What this means in practice
- •For voice assistant developers: Improve speech recognition accuracy in noisy or far-field conditions by reducing incorrect extra words during transcription.
- •For meeting transcription service teams: Reduce false insertions in transcriptions of complex, multi-speaker audio recordings to produce more reliable meeting records.
Authors
Tingzhen Xiong, Rilin Chen, Weiwei Li, Wentao Zhang, Qicong Xie
Abstract
When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.