Unified speech model improves voice restoration and speaker isolation
XingChen-UniFE: A Unified Generative Speech Front-End via Complementary Acoustic-Semantic Modeling
Sound
Summary
Cleaning up unclear or noisy speech and separating one person's voice from others are common challenges. The authors present a single model that tackles both problems by combining detailed sound patterns with high-level language features. This approach helps the model to better understand and rebuild speech, even when the audio is heavily distorted or unfamiliar. Tests show the model works well for different tasks and tough conditions.
What this means in practice
- •For audio engineers: Enhance recordings by restoring damaged speech and isolating target speakers using a single model that integrates semantic and acoustic information.
- •For telecommunications operators: Improve call quality by automatically removing noise and extracting main speakers in real-time from complex audio streams.
Authors
Kelan Kuang, Qihan Hu, Jian Kang, Jie Li, Yongxiang Li, Yi Liu
Abstract
This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.