Multimodal model predicts diverse brain responses to stimuli accurately
MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching
Artificial Intelligence
Summary
The usual way to predict brain activity from images or sounds assumes the brain will respond the same way every time, but actually, brain responses vary due to many factors. The authors created MultiFlow, a new model that predicts not just one brain response but a range of possible responses, capturing uncertainty and variation. It uses multiple types of input like vision, audio, and language, combining them with a special mathematical approach called flow matching. MultiFlow matches the accuracy of large groups of models but with just one, helping understand how the brain reacts to different stimuli more realistically.
What this means in practice
- •For brain imaging teams: Generate more accurate and diverse predictions of brain activity from visual and auditory stimuli in single-subject fMRI studies using one model.
- •For multimodal ai developers: Integrate stimulus-conditioned flow matching to improve uncertainty estimates and sample diversity when modeling complex sensory inputs.
Authors
Bui Van Hung, Hoang-Son Vo, Mai Duc Minh Huy, Dang Thanh Vu, Soo-Hyung Kim
Abstract
Current fMRI encoding models frame prediction as deterministic regression, yet identical stimuli evoke a distribution of neural responses (arising from physiological noise, hemodynamic variability, and cognitive fluctuations) that a single point estimate cannot capture. This work introduces MultiFlow, a generative fMRI encoding framework based on Conditional Flow Matching (CFM) and a formulation termed Stimulus-Informed Source Flow Matching (SISFM). SISFM generalises CFM by conditioning the source distribution on the stimulus: a learnable temporal convolution shapes the source mean with an HRF-length receptive field, while a variance head shrinks its scale, shortening the expected transport path. SISFM is instantiated with a Diffusion Transformer (DiT-X) decoder parameterising the velocity field, with sub-layer Attention Residuals, heterogeneous fusion of ten frozen vision, audio, language, and audiovisual backbones, and per-subject Yeo-network heads. On the Algonauts 2025 dataset, a single MultiFlow checkpoint recovers most of the benefit of large deterministic ensembles: it trails the 1000-model TRIBE ensemble in-distribution by 0.003 PCC (0.3163 vs. 0.3195), and the strongest in- and out-of-distribution ensembles (VIBE-20, MedARC-50) by 0.010 and 0.009 PCC, using a single trained model rather than an ensemble of 20-1000. The same checkpoint also provides stimulus-conditioned uncertainty estimates and sample-based diversity.