Self supervised speech features improve restoration when combined early
When Do Self-Supervised Representations Help Speech Restoration?
Sound
Summary
Speech restoration tries to fix noisy or broken audio recordings. The authors found that trying to clean up the speech features used by AI models before restoring often hurts performance when the noise is very bad. Instead, they propose a method that shares partly restored speech information early on with the feature system, helping it better understand and fix the noise. This approach improves speech restoration quality compared to standard methods.
What this means in practice
- •For speech technology developers: Build better speech restoration systems by integrating early restored features with self-supervised models to improve audio quality in noisy environments.
- •For audio forensics teams: Enhance the clarity of degraded speech recordings by using early restoration injection techniques to guide self-supervised feature encoders.
Authors
Yongjoon Lee, Younghoo Kwon, Jung-Woo Choi
Abstract
Self-supervised learning (SSL) representations are widely used in speech restoration (SR), and the prevailing trend is SSL feature cleansing: fine-tuning the SSL to recover clean features from degraded speech, then conditioning the restoration model on them. However, the representational capability of SSL when encoding degraded speech has been underexplored, and we ask whether cleansing always benefits SR training. Across SSL encoders and perceptual quality proxies, we discover perceptual correlation collapse---under severe degradation, cleansing degraded SSL features toward clean ones misaligns with the SR objective. We hypothesize the restoration backbone can supply the SSL with clues to resolve this collapse, since its intermediate layers already produce partially restored features. We therefore propose Early Restoration Injection (ERI), which routes intermediate features from the restoration backbone into the SSL through a lightweight conditioning network. ERI excels over the unconditioned baseline and, on the VoiceFixer benchmark, achieves competitive performance among multiple baselines.