Neurons++ reconstructs images and videos from brain activity signals
Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability
Computer Vision and Pattern Recognition
Summary
It is difficult to understand what people see by looking at their brain activity. The authors created Neurons++, a tool that reconstructs both still images and moving videos from fMRI brain scans. This method organizes brain signals into different parts like space, meaning, language, and time to create better pictures and videos. Their system works better than previous ones in video reconstruction and performs well for images, and it can also adjust outputs based on specific brain areas.
What this means in practice
- •For brain-computer interface developers: Build systems that decode both images and videos from fMRI scans for more natural visual communication.$Commercial implications: Enables advanced brain-computer interface devices to visualize user perception for communication or interaction purposes.
- •For medical imaging teams: Use improved fMRI decoding to better understand patients’ visual perception and cortical function based on both images and videos.
Authors
Haonan Wang, Yuchen Wang, Jingyu Lu, Qixiang Zhang, Lehan Wang, Xuanqi Huang, Xiaomeng Li
Abstract
Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.