Fine-tuning language models improves automatic speech correction accuracy
Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses
Computation and LanguageArtificial Intelligence
Summary
Automatic speech recognition (ASR) sometimes makes mistakes in transcribing spoken words. The authors show how teaching large language models to pick the best transcription or fix errors from multiple options can reduce mistakes more effectively than just prompting the models. They do this by training on carefully chosen examples where the correct transcription is present, absent, or already correct, helping the model learn when to rewrite or keep the original. This approach works best with models around 3 billion parameters and in certain types of speech data, revealing challenges in publicly available datasets.
What this means in practice
- •For speech recognition developers: Improve ASR transcript accuracy by fine-tuning language models to selectively rewrite or choose among multiple transcription options.
- •For call center technology teams: Enhance transcription quality in voice-to-text tools by reducing errors from automatic recognition systems using model-guided post-editing techniques.
Authors
Shiran Liu, Gaohui Xu, Sergio Burdisso, Thibault Bañeras-Roux, Shashi Kumar, Petr Motlicek
Abstract
Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer. However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models. We propose oracle-guided fine-tuning that turns N-best lists into supervision via three curated pools: corrections (reference in the list but not top-1) to teach selection, regularizers (top-1 correct) to discourage unnecessary rewrites, and hard negatives (reference absent) to suppress hallucination. Fine-tuning on this data provides more effective supervision than random subset fine-tuning for learning a hybrid select-or-rewrite policy, with the clearest gains from around 3B parameters on the proprietary dataset but less conclusive results on CommonVoice, a divergence suggesting that public read-speech text may overlap with LLM pretraining and mask genuine post-editing gains.