MindVLM turns brain activity into personalized image descriptions
MindVLM: Neural-Grounded Visual Captioning via Subject-Aware Semantic Evidence Selection
Computational Engineering, Finance, and Science
Summary
Translating what people see in their brains directly into words is hard because brain data and text descriptions don’t always match. The authors found that existing methods use captions describing whole images, but brain scans often reflect only parts of an image someone is looking at. To fix this, they created MindVLM, which uses brain signals to select visual cues before generating captions, making the output more personalized and accurate. This method works without relying on direct text training and performs well compared to traditional captioning systems.
What this means in practice
- •For neurotechnology developers: Create brain-computer interface tools that translate visual brain signals into personalized natural language descriptions without needing paired text training data.$Commercial implications: Enables development of consumer or clinical brain-to-language communication devices by providing scalable neural decoding without reliance on extensive text-labeled datasets.
- •For assistive technology engineers: Build visual aids that convert a user’s brain responses to viewed scenes into understandable descriptions to assist people with communication impairments.
Authors
Zixiang Yin, Yu-Ping Wang, Zhengming Ding
Abstract
Translating visual brain activity into natural language offers a powerful window into human cognition and neural semantic representation. Most prior brain-captioning approaches project fMRI signals into language space using paired supervision. However, the widely used Natural Scenes Dataset (NSD) was built based only on image-fMRI pairs, while existing methods rely on MS-COCO captions as indirect textual supervision. Critically, during fMRI acquisition, participants observe only cropped image regions, whereas the associated MS-COCO captions describe full scenes. This mismatch creates a fundamental gap between neural representations and textual supervision, introducing an inherent semantic loss in NSD-based brain captioning. To mitigate this limitation, we reformulate brain captioning as Neural-Grounded Visual Captioning, where language is generated directly from visually grounded neural evidence rather than text supervision. We propose MindVLM, a framework that maps fMRI signals into subject-aware visual tokens and leverages a frozen vision language model (VLM) to produce interpretable and personalized descriptions. Instead of point-wise regression, MindVLM focuses on extracting semantically relevant visual cues from neural activity, naturally accommodating partial observability and inter-subject variability. Extensive experiments on NSD show that MindVLM consistently improves caption quality across multiple VLM backbones, including LLaVA-v1.5, Qwen2.5-VL, and Gemma3, beyond intrinsic model priors. Despite requiring neither text supervision nor an LLM during training and operating under an image-grounded inference paradigm distinct from conventional captioning methods, MindVLM remains competitive with supervised approaches, suggesting that subject-specific neural grounding provides an efficient, scalable, and interpretable path toward faithful brain-to-language generation.