Audio helps large language models fix speech errors without breaking correct words
Text-Only LLM Correctors Break Correct Transcripts: A Repair-Damage Analysis of Audio Grounding in ASR Error Correction
SoundComputation and Language
Summary
Speech-to-text models sometimes make mistakes, and large language models (LLMs) can help fix these errors. The authors find that when LLM correctors work only with text, they often break words that were already correct. But when the correctors also have access to the original audio, they make fewer mistakes and fix more errors. This shows that using the audio along with text helps improve speech transcription corrections.
What this means in practice
- •For speech technology developers: Improve speech-to-text correction systems by integrating original audio input to reduce errors introduced during text-only correction.
- •For medical transcription teams: Enhance medical dictation accuracy by balancing error repair and insertion management when applying audio-informed LLM corrections.
Authors
Vasiliy Kudryavtsev, Kirill Borodin, Nikita Vasiliev, Mikhail Gorodnichev, Grach Mkrtchian
Abstract
LLM correctors of ASR output are judged by aggregate WER, which hides what they fix and what they break. We decompose every WER change exactly into repaired tokens, damaged tokens and insertions, and use this to ask what the source audio adds to a corrector. We compare correction with and without the audio, for the model's own draft and for the output of an independent recogniser, on three corpora. With Gemini 2.5 Flash, text-only correction of its own draft breaks 13-32% of already-correct transcripts; with the audio, 1-3%. Damage falls in both settings. Repair falls against the own draft and rises against the recogniser. On medical dictation, kept insertions cancel the gain in net WER. The open-weight Voxtral, which rarely edits, does not show this pattern. Prompts differ between the two arms, so the contrasts bound the effect of the audio and do not isolate it.