Speech enhancement improves with one-step dual latent drifting approach
DriftSE: Speech Enhancement with Generative Drifting
SoundArtificial Intelligence
Summary
Cleaning up noisy speech recordings is important for clearer communication and better voice recognition. The authors created a method called DriftSE that improves speech clarity in just one step by using two kinds of hidden information about sounds: one that focuses on meaning and one that focuses on the actual physical sound details. This approach works well even without examples of noisy and clean speech pairs during training and can be used in real-time applications. Tests show that DriftSE leads to better word recognition accuracy than previous methods.
What this means in practice
- •For voice assistant developers: Improve real-time speech input clarity using one-step enhancement that preserves both phonetic and acoustic details without needing paired noisy-clean data.
- •For telecommunications engineers: Enhance call quality by reducing noise and reverberation efficiently with a model adaptable to various generator architectures and datasets.
Authors
Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan, Rasmus Kongsgaard Olsson
Abstract
We propose DriftSE, a novel one-step generative framework for speech enhancement formulated as a latent distribution equilibrium problem. During training, the drifting field aligns the generator's pushforward distribution with the clean speech manifold through drifting in a latent domain. During inference, the drifting process is discarded, enabling one-step generation. We establish that its enhancement quality depends fundamentally on the choice of latent representation. Semantic latents preserve phonetic structure but fail to capture physical acoustic cues, whereas acoustic latents reconstruct the physical signal but risk linguistic hallucination. Therefore, we introduce dual-latent drifting, performing parallel drifting in both semantic and acoustic latents to simultaneously preserve phonetic intelligibility and acoustic fidelity. Additionally, we demonstrate that DriftSE enables fully unpaired training by aligning latent distributions rather than exact point-wise targets. Consequently, DriftSE facilitates cross-dataset learning in the absence of paired noisy-clean samples. Moreover, DriftSE exhibits broad architectural flexibility across different generator backbones. Extensive evaluations on additive denoising and convolutive dereverberation demonstrate robust one-step enhancement across both offline and real-time causal settings. Notably, DriftSE achieves state-of-the-art word error rates across all four evaluated datasets while strictly operating at 1 NFE. Code and audio examples are available online.