EEG signals converted into images using wavelet cross-attention and flow matching
E2I-Flow: EEG-to-Image Reconstruction via Wavelet-Temporal Cross-Attention and Conditional Flow Matching
Computer Vision and Pattern RecognitionMachine Learning
Summary
Decoding pictures from brainwaves recorded via EEG is hard because EEG signals have complex patterns over time and frequency. The researchers created E2I-Flow, a method that learns important features of these brain signals using special wavelet and time-based processing combined through an attention mechanism. Then, it uses a flow-based model conditioned on these features to generate image representations that match the original visuals. This approach improved the accuracy of retrieving and reconstructing images from brain signals compared to prior methods.
What this means in practice
- •For brain computer interface developers: Generate visual content directly from EEG signals to enhance communication devices for users with limited motor abilities.$Commercial implications: Enables new EEG-based image decoding products for assistive communication and neurotechnology markets.
- •For neurotechnology engineers: Improve brain signal interpretation by combining time-frequency EEG features and flow models to decode visual information more accurately.
Authors
Haowei Pan, Hwa Hui Tew, Junn Yong Loo, Fuad Noman, Hernando Ombao, Chee-Ming Ting
Abstract
Electroencephalography (EEG)-based visual decoding and reconstruction aims to recover visual information from non-invasive brain signals. But there are challenges due to the complex temporal-frequency characteristics of EEG signals and the difficulty of discrete denoising framework in mapping neural activity to meaningful visual representations. Here, we propose E2I-Flow, an EEG-to-image decoding and reconstruction framework that combines a novel EEG encoder for time-frequency representation learning from EEG features, and conditional flow matching for visual decoding. A stationary-wavelet frequency branch and a temporal-mixing branch are integrated through bidirectional cross-attention and adaptive fusion, with the resulting EEG representations aligned to ground-truth image embeddings using contrastive and mean-squared-error loss supervisions. The aligned EEG representation is then used as a condition for a Transformer-based flow-matching model to predict the corresponding visual embedding for image synthesis. Experimental results show strong zero-shot EEG-to-image retrieval and improved visual reconstruction performance, outperforming several state-of-the-art baselines.