Neural model reproduces visual brain activity for chart viewing
Can a Neural Encoding Model Replicate an fMRI Visualization Study?
Human-Computer InteractionComputer Vision and Pattern Recognition
Summary
Understanding how our brains see graphs usually comes from watching how people behave, not directly from brain scans, which are hard to do. The authors tested a computer model called Tribe V2 to see if it could mimic brain activity patterns seen in a past brain imaging study about how people look at bubble and 3D surface charts. The model predicted brain responses that matched most of the effects found in the real brain data, especially in areas related to vision. This means the model shows promise for simulating brain responses to visual information but does not independently prove the original brain findings.
What this means in practice
- •For visualization developers: Generate predicted brain responses to new visualizations to guide design choices without costly fMRI studies.
- •For machine learning engineers: Use Tribe V2 to simulate neural data for training or testing AI models that link brain activity and visual stimuli.
Tested on one dataset.
Authors
Erfan Nasirzadeh Orang, Zack While
Abstract
Most knowledge of graphical perception comes from behavioral studies. Understanding from a neural perspective is much more limited due in part to neuroimaging studies' expensiveness and difficulty to conduct. In this paper, we evaluate whether Meta's Tribe V2 neural encoding model can recover neural contrasts from a visualization fMRI study. Specifically, we evaluate Tribe V2 through a conceptual replication of the visualization-viewing component of a prior comparison of Bubble charts and three-dimensional Surface charts in color and grayscale. We generate TRIBE-predicted cortical responses for the original stimuli and compare the resulting contrasts with those reported in the human study. The model reproduced the direction of 11 of 14 reported cortical effects, with agreement concentrated in visual-processing regions. This agreement characterizes the model's alignment with the prior human-generated fMRI results rather than independently confirming them. We discuss the limitations encountered when working with this model for in-silico replication and hope to encourage future work exploring this new avenue for neuroimaging studies in visualization. Supplemental materials are available at https://osf.io/8a96x/.