Self-supervised learning aligns brain scans across people for image decoding
Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI
Machine LearningArtificial Intelligence
Summary
Brain scans can show how people see images, but comparing scans from different people is hard. The authors use a method called self-supervised learning that trains brain scan models without using the actual images shown to people. Their approach creates brain data representations that work well to describe and recreate images from scans. Remarkably, models trained on one person’s brain data can be matched to another person’s data without shared images, allowing decoding tools to work across individuals.
What this means in practice
- •For neuroimaging analysts: Combine brain scan data from multiple participants without shared stimuli to improve cross-subject decoding models.
- •For machine learning engineers: Build models that predict sensory-evoked brain responses without access to stimulus images, enabling new kinds of neural decoding pipelines.
Tested on one dataset.
Authors
Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann
Abstract
Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.