Brain activity decoded into shared visual semantic space across datasets
Many Brains, One Geometry: A Shared Visual-Semantic Space for Cross-Dataset fMRI Decoding
Human-Computer Interaction
Summary
Brain scans that record visual activity usually analyze each person and experiment separately. This paper introduces a method called BRAID-fMRI that can align brain responses from many people and different experiments into one common space using a special image-language AI called CLIP. This shared space helps decode what people see more accurately across diverse brain data. The authors show that using data from many individuals greatly improves the ability to identify the viewed images from brain signals.
What this means in practice
- •For brain-computer interface developers: Decode visual experiences from brain activity using a single model trained on diverse datasets and participants for more general and robust results.
- •For neurotechnology companies: Develop cross-individual cognitive decoding tools by aligning brain signals into a common semantic space to improve device accuracy and user adaptation.$Commercial implications: Enables more versatile brain-computer interfaces that work across many users, facilitating commercial neurotech products.
Authors
Moein Khajehnejad, Michelangelo Tronti, Forough Habibollahi, Tommaso Boccato, Matteo Ferrante, Nicola Toschi
Abstract
Visual decoding from fMRI is typically siloed by participant and experiment, obscuring whether heterogeneous neural measurements can be organized within a common computational geometry. Here we introduce BRAID-fMRI (Brain Representation Alignment across Individuals and Datasets), a shared CLIP-supervised decoding framework. BRAID-fMRI uses a single ROI-wise Transformer with optional participant conditioning across eight visual-fMRI datasets comprising 93 dataset-specific participant entries, 430,007 single-trial responses and 162,839 unique stimuli. Regional brain activity is aligned with 512-dimensional CLIP ViT-B/32 representations using a multi-positive contrastive objective that treats repeated stimuli across participants and datasets as positives. BRAID-fMRI supports retrieval across seven evaluation datasets. On eight matched participant entries, it achieves 35.0 +/- 11.1% Top-10 accuracy, exceeding the observed mean accuracy of the two evaluated baselines - the MindEye-style pooled-CLIP decoder (27.1 +/- 5.1%) and ridge regression (21.1 +/- 9.1%) - and attaining the highest observed accuracy for seven of eight entries. In separately trained participant-agnostic models, expanding the source pool increased target-dataset holdout accuracy by up to 92.1% relative to the initial source-training condition. The learned space preserves graded semantic structure, while ablations and saliency highlight ventral and early visual cortex and category-specific motion and attentional systems. These results support scalable cross-dataset decoding into a common CLIP-aligned space, with model sensitivity concentrated in ventral and early visual inputs.