NeuroCodec improves high-fidelity speech extraction from EEG signals
NeuroCodec: EEG-Guided Latent-Domain Generative Model for High-Fidelity Target Speaker Extraction
Machine Learning
Summary
It is hard to focus on one speaker when many people talk at once. This paper introduces NeuroCodec, a system that uses brain signals recorded with EEG to help computers isolate someone’s speech from noisy backgrounds with high quality. Unlike older methods that blur sounds together, NeuroCodec works in a special compressed sound space, using advanced neural models to better understand brain and audio data together. Tests show it sounds clearer and more natural, even in real-time settings useful for hearing aids.
What this means in practice
- •For hearing aid developers: Create hearing aids that use EEG signals to extract a target speaker’s voice with high clarity in noisy multi-speaker environments.$Commercial implications: Enables next-generation hearing aids combining EEG-based attention decoding with high-fidelity speech extraction for better user experience.
- •For audio signal processing engineers: Develop real-time streaming systems that separate overlapping speech signals guided by neural recordings for improved speech enhancement tools.
Authors
Julian Silva, Jaliya Kodithuwakku, Navindu Sirimanna, Shaveen Herath, Wageesha N. Manamperi, Joshua P. Kulasingham
Abstract
Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.