Codec token method improves voice extraction from mixed audio
What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective
Sound
Summary
Extracting one person's voice from a mix of sounds is hard because speech contains meaning, speaker identity, and sound details all mixed together. The authors found that separating these factors into two kinds of tokens—one for what is said and another for how it sounds—helps improve this task. They designed a system where one part focuses on understanding meaning related to the target speaker and another part recreates the sound, working together to produce clearer speech. Their experiments show their method reduces errors significantly while keeping good sound quality and speaker likeness.
What this means in practice
- •For voice assistant developers: Improve voice assistant accuracy by better extracting the user’s speech from noisy environments using disentangled semantic and acoustic token modeling.
- •For telecommunications engineers: Enhance call clarity in multi-speaker scenarios by applying semantic-acoustic disentangled tokens to isolate a speaker's voice from overlapping speech.
Authors
Wang Xiang, Bang Zeng, Ming Li
Abstract
Target speaker extraction (TSE) aims to recover a target speaker's speech from a speech mixture using an enrollment utterance. Recent codec-based generative approaches formulate TSE as conditional speech token generation and achieve promising perceptual quality. However, most existing systems rely on unified codec tokens that entangle linguistic content, speaker characteristics, and acoustic details, forcing a single model to jointly handle target selection, content semantic, and acoustic reconstruction. In this paper, we investigate whether disentangled semantic-acoustic codec representations are better suited to TSE. We first show that simply replacing unified codec tokens with semantic-acoustic tokens is insufficient and can even degrade performance when both remain handled by a unified generative model. Motivated by this observation, we propose a task-aware discriminative-generative framework, where a discriminative branch extracts target-related semantic information, while a generative branch predicts coarse acoustic tokens and refines them under semantic guidance. This specialization separates what is said from how it sounds while coordinating both factors in a factorized codec space. Experiments on Libri2Mix show that our method reduces dWER from 0.388 to 0.118 compared with a unified-codec baseline, while maintaining competitive perceptual quality and speaker consistency. Ablation studies confirm that the gains arise from task-aware semantic-acoustic modeling rather than representation disentanglement alone.