Phonology aware system improves speaker extraction in mixed language audio
Phonology-Aware Speaker Extraction Frontend for Audio LLMs
Sound
Summary
Understanding a specific speaker's voice in a noisy room with many people talking is hard, especially when languages mix. The authors developed LATTE, a small smart tool that decides how much work to do depending on the difficulty of the audio. It uses knowledge about speech sounds and can adapt without knowing the language in advance. LATTE performs almost as well as bigger systems but uses much less computing power, helping machines better recognize mixed voices quickly.
What this means in practice
- •For speech recognition engineers: Implement a frontend that efficiently extracts a target speaker from overlapping multilingual audio while reducing computational costs.
- •For telecommunication system developers: Integrate an adaptive speaker extraction module to improve call clarity and transcription accuracy in noisy, multilingual environments.
Authors
Yanxiang Chen, Haoyuan Deng, Jinyu Wang, Hongyi Chen, Dingshan Li, Runlin Huang, Weifeng Su
Abstract
Target speaker extraction (TSE) is a key frontend for enabling audio large language models (LLMs) to recognize and understand a designated speaker in overlapping speech. However, multilingual mixtures exhibit heterogeneous extraction demands, causing fixed-depth systems to either under-process challenging inputs or waste computation on inputs that have already converged. Existing TSE frontends generally apply uniform computation, while Audio-LLM adaptation methods neither explicitly model language-conditioned extraction demands nor preserve plug-and-play compatibility with frozen downstream models. We therefore propose LATTE, a lightweight 5.2M-parameter frontend that combines an acoustic Difficulty Estimator, a task-adaptive phonology-informed prior learned from multi-depth TSE gains, and an Adaptive Hourglass Extractor to route each mixture without requiring language identity at inference. On PolyExtract, LATTE achieves a macro SI-SNRi of 15.07 dB at 0.30 GFLOPs, retaining performance within 0.09 dB of a 0.53-GFLOP fixed-depth model while reducing computation by 43.4%, and lowers the average WER from 16.8% to 14.6% relative to TF-GridNet. By jointly modeling sample-level extraction demands and phonological structure, LATTE provides a practical framework for quality-aware, compute-efficient multilingual TSE and offers a scalable path toward robust Audio-LLM deployment.