Towards Robust Generative Speech Enhancement Using Vector Quantisation-Based Neural Audio Codec
2026-06-15 • Sound
Sound
AI summaryⓘ
The authors studied two ways to improve speech enhancement using neural audio codecs: one using continuous internal signals (cNAC-SE) and one using discrete tokens (dNAC-SE). They analyzed how these methods work and found that the continuous approach generally works better in different test scenarios. They also discovered that vector quantisation (VQ) helps make the models more robust by acting as a type of regularisation, which can help even when using continuous methods. This suggests that the benefits of VQ go beyond just handling discrete data.
neural audio codecspeech enhancementvector quantisationlatent spacecontinuous representationsdiscrete tokensregularisationDNS-MOS metrics
Authors
Haixin Zhao, Nilesh Madhu
Abstract
This work investigates modelling strategies in continuous and discrete latent spaces in the vector quantisation (VQ)-based neural audio codec (NAC) speech enhancement (SE), along with the role of VQ regularisation. We propose cNAC-SE and dNAC-SE frameworks that predict continuous representations and discrete tokens in latent space, respectively. Theoretical analysis and visualisations in latent space are performed to exhibit their inherent modelling mechanisms. Experimental results show that the fully fine-tuned cNAC-SE model consistently outperforms all dNAC-SE variants across diverse test conditions and achieves leading performance among established generative approaches in DNS-MOS metrics. Comparison with the discriminative counterpart shows that VQ enhances robustness through an intrinsic effect of clean-prior-constrained regularisation, independent of discrete token processing. This highlights the transferable value of VQ regularisation to other continuous modelling methods.