Speech enhancement model improves transcription accuracy on real recordings

Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching

Machine Learning

Summary

Speech enhancement systems aim to improve the clarity of speech recorded in noisy environments. The authors show how to better train these systems using real noisy recordings and just their transcriptions, without needing clean reference audio. They adapt a technique called Reinforce Adjoint Matching to guide the model toward reducing speech recognition errors. Their improved model lowers the word error rate on real recordings without making the enhanced speech sound worse.

What this means in practice

  • For speech technology engineers: Improve speech recognition accuracy by post-training enhancement models using real noisy recordings and transcripts without needing clean audio targets.
  • For call center operators: Enhance customer call audio quality and transcription accuracy in real-world noisy conditions without extensive labeled clean data.$Commercial implications: This approach enables improved automated transcription services for call centers leveraging real recordings, reducing labeling costs and improving service quality.

Authors

Julius Richter, Christoph Boeddeker, Yoshiki Masuyama, Kohei Saijo, Dominik Klement, Gordon Wichern, Jonathan Le Roux

Abstract

We adapt Reinforce Adjoint Matching (RAM), a reward-based post-training method, to generative speech enhancement (SE). Starting from a pretrained SE model, RAM tilts the model's conditional distribution toward outputs with higher reward. During training, the current model generates enhanced speech on-policy, evaluates each generated endpoint with a potentially non-differentiable reward, and analytically re-noises the endpoint to construct inputs for a reward-guided regression objective. This enables post-training directly on real recordings using weak supervision, such as text transcripts, without requiring paired clean speech targets or reward gradients. We investigate word error rate (WER)-based post-training and whether recognition performance can be improved without compromising perceptual speech quality. Experiments on real CHiME-4 recordings reduce WER by 5.08 percentage points relative to pretrained FlowSE without reducing any of the reported non-intrusive speech quality metrics. A subjective listening test at the default reward scale finds no statistically significant preference between the post-trained and pretrained models.