Speech enhancement speeds up real world noise removal with unpaired learning
SE-MSB: End-to-End Unpaired Speech Enhancement using Mamba Schrödinger Bridges
Artificial IntelligenceSound
Summary
Speech enhancement models usually need clean and noisy speech pairs to learn how to improve sound quality, which can be limiting in real situations. The authors propose a new method that learns from separate collections of clean and noisy speech without pairing them. Their approach uses a special mathematical process called Diffusion Schrödinger Bridges combined with a fast Mamba Diffusion Model to improve efficiency. Their results show the method performs as well or better than current models while running much faster. It can also adapt well to different speech enhancement tasks, making it useful for a variety of real-world noise problems.
What this means in practice
- •For audio software developers: Create speech enhancement features that improve noisy recordings without needing matched clean-noisy pairs, enabling faster real-time processing.
- •For telecommunications engineers: Develop robust noise reduction systems that generalize well across different acoustic environments for clearer voice communication.
Authors
Andreas Bagge, Andreas Nymand, Michael Riis Andersen, Bjørn Sand Jensen
Abstract
Speech enhancement (SE) models typically rely on supervised learning with paired data examples where clean speech is synthetically degraded. This paradigm limits performance in real-world scenarios where the target environment's specific acoustic characteristics are unknown. We propose a fully unpaired SE framework that uses principled Diffusion Schrödinger Bridges (DSB) to learn a stochastic transport process between a clean and a degraded speech distribution. Algorithms for learning transport maps are computationally heavy since they require simulating differential equations during training, usually at each training step. Therefore, we propose using a high-efficiency Mamba Diffusion Model designed for end-to-end waveform processing. We compare against state-of-the-art methods for speech enhancement, both paired and unpaired, as well as a classical signal processing algorithm. Experimental results show that we are on par or better than the baselines while being orders of magnitude faster during inference. Furthermore, we show that the flexibility of the DSB formulation allows our model to generalize across SE tasks, offering a robust and efficient solution for real-world speech restoration.