Multimodal brain model lacks visual and auditory spatial maps but shows strong motion signals
An In-Silico Audit of Topographic Organization in a Multimodal Brain-Response Model
Artificial Intelligence
Summary
This paper checks whether a computer model that predicts brain activity from videos, sounds, and text matches known brain maps that show where things appear in our sight or hearing. The authors found the model does not capture how the brain represents specific locations in space for sight or sound. However, it does accurately reflect parts of the brain that respond to motion. This means the model is good for studying features but not for understanding brain maps that depend on where things appear in space or frequency.
What this means in practice
- •For machine learning engineers: Use the model to analyze feature-based brain responses where spatial location is not critical.
- •For neuroimaging software developers: Avoid relying on this model for simulations requiring spatial maps of visual or auditory cortical areas.
Authors
Jakub Strawa
Abstract
TRIBE v2 (d'Ascoli et al., 2026) predicts fMRI responses across the cortical surface from video, audio and text. Its authors validated it with category-selective localizers (faces, places, bodies, words) and language contrasts, but not with the classic topographic organization of sensory cortex: retinotopy, tonotopy, and motion selectivity in area MT+. There is a clear architectural reason to expect retinotopy to fail: TRIBE's video features are averaged over space before the response head, which should discard visual-field position. I present mapping stimuli, a fitting pipeline adapted from standard fMRI mapping analyses, and a stand-in model with known ground truth that serves as a positive and negative control, and apply them to TRIBE v2's predictions, comparing the fitted maps with human atlases using spin tests. TRIBE v2 shows a sharp split. There is no contralateral hemifield response and no polar-angle map, robustly across stimulus controls. There is no tonotopic gradient in primary auditory cortex: best frequencies span an interquartile range of only 0.14 octaves across core and belt, and the null persists with loudness-equalized tones. An eccentricity-like gradient appears only when the ring geometry changes, consistent with global image statistics rather than a positional code. In contrast, motion selectivity is strong and precisely localized: 22% of the 1% most motion-selective vertices fall in MT+, which covers 0.42% of cortex. In-silico experiments that depend on feature identity appear viable with TRIBE v2, while those that depend on visual-field or frequency position are not supported by the current model.