Whisper speech recognition reduces false transcriptions on silence
How to Reduce Whisper Hallucination
Computation and Language
Summary
Speech recognition models sometimes make up words when there is no speech, like in silence or background noise. This paper shows that current versions of Whisper, a popular speech model, often hallucinate words in silence. The authors improve this by creating a new dataset of fake speech phrases made with a text-to-speech system, teaching the model to recognize both real and fake speech. Their method greatly lowers false words in silent clips while improving the model’s ability to correctly recognize real speech.
What this means in practice
- •For speech recognition developers: Improve transcription accuracy by reducing false word detection on silent or non-speech audio in multilingual models.
- •For voice assistant engineers: Enhance voice detection systems to avoid triggering on silence or background noises while maintaining correct recognition of spoken commands.
Authors
Husein Zolkepli
Abstract
Whisper is still what runs in production: one permissively licensed checkpoint, 99 languages, no per-language tuning. But it writes sentences nobody said. On 42 clips of pure room tone, whisper-large-v3 emits words on 61.9% of them and emits something on 100%. The usual response is to distil a student from Whisper pseudo-labels, filtering the hallucinations out of the corpus first, which deletes the evidence while leaving the behaviour in place: the student is fitted to the subset where the teacher was right, and inherits a failure mode absent from its own training data. The second response, adding non-speech audio so the model learns to stay quiet, already ships and works, but teaches suppression without discrimination. The checkpoint best on every non-speech arm here is also the one that recovers the fewest genuinely spoken phrases and deletes 58% of repeated speech. The teacher has to be fixed, with both halves of the signal. Our benchmark scores both at once: 11,852 clips over eight arms, 6,267 synthetic positives, 8,296 clips of real audio that made a production model fail, and FLEURS in 58 languages. We collect 40,891 hallucination phrases in 100 languages, choose a text-to-speech system by measurement, and synthesise those phrases as positives, so the model meets the same text both as something to suppress and as something to transcribe. Across 33 matched pairs of fine-tunes differing only by those positives, adding them lowers word emission on real voice-free audio in 31 pairs and raises phrase recovery in 32; among the 30 pairs that hold English accuracy it is 30 out of 30. The best checkpoint takes hallucination on silence from 61.9% to 2.4% and words over real voice-free audio from 99.9% to 47.8%, while raising phrase recovery from 69.8% to 82.7%. The benchmark, lexicon and synthetic corpus are released at https://huggingface.co/datasets/Scicom-intl/Whisper-Hallucination