Ramamba net improves detecting focus of hearing in noisy places
RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection
Artificial Intelligence
Summary
People trying to hear one speaker in noisy places use brain signals to detect who they are focusing on. EEG brain signals alone miss some important clues, so this paper looks at combining EEG with eye movement signals (EOG) for better detection. The authors created RAMamba-Net, which mixes these signals in a smart way that pays attention to how reliable each signal is at each moment. Their system improves accuracy and is more robust when signals are noisy or change. This might help devices like hearing aids understand what users want to hear more reliably.
What this means in practice
- •For hearing device developers: Build hearing aids that better identify which speaker a user is focusing on by combining EEG and EOG signals with reliability-aware fusion.$Commercial implications: Enables advanced hearing aids to decode user auditory attention more accurately, improving user experience in noisy environments.
- •For human computer interface engineers: Create systems that track user auditory focus more robustly by integrating brain and eye signals with adaptive model weighting.
Authors
Xingyi He, Ziwei Wang, Dongrui Wu
Abstract
Auditory attention decoding (AAD) identifies the attended speaker from physiological signals, supporting neuro-steered hearing devices and natural human-machine interaction. Electroencephalography (EEG) is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes, motivating EEG and electrooculography (EOG) fusion. Existing approaches remain limited by weak cross-modal interaction, inefficient temporal modeling, and low robustness to sample variations. To address the limitations, we propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD. RAMamba-Net employs a Mamba-enhanced band-aware convolutional Transformer to capture band-specific EEG patterns and long-range temporal dynamics. A dual-branch temporal-spatial encoder models EOG temporal and inter-channel dependencies. Cross-modal attention enables explicit modality interaction. Then, a reliability-aware module is introduced to estimate sample-wise modality weights for feature and prediction consistency, thereby enhancing multimodal fusion. Experiments on two AAD benchmarks demonstrate that RAMamba-Net effectively exploits complementary EEG-EOG information, yielding accuracy gains of 5.76% over unimodal baselines, together with more robust decoding and discriminative representations. Further analyses show that explicit cross-modal interaction improves multimodal alignment, while the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.