Brain models better match people’s picture similarity judgments
Alignment of visual brain encoding models with individual perceptual similarity judgments
Computer Vision and Pattern Recognition
Summary
Brain models used to predict how images activate the brain don’t always match how individual people see similarities between pictures. The authors fine-tuned these models using each person’s own similarity judgments on pictures. This personalized tuning made the brain models reflect individual perception more closely than before. It also captured how people differ or agree in their perception better than generic models.
What this means in practice
- •For neurotechnology developers: Create brain-computer interfaces that adapt to individual visual perception by fine-tuning brain response models using personal similarity judgments.$Commercial implications: Enables personalized neurotechnology products that better interpret brain responses to images, improving user-specific applications like communication aids.
- •For computer vision engineers: Improve visual encoding models by integrating personalized human similarity data to better predict how individuals perceive images.
Tested on one dataset.
Authors
Daniel Chong, Haomiao Chen, Keith W Jamison, Raja Marjieh, Ilia Sucholutsky, Nori Jacoby, Amy Kuceyeski
Abstract
Encoding models that predict brain activation responses to images have traditionally been trained to maximize accuracy when predicting observed brain data. However, their internal representations may not reflect individual human perception. In this work, we investigate whether we can fine-tune brain encoding models to better align their representations with individuals' similarity judgments. Using similarity judgments of 100 images from 8 participants in the Natural Scenes Dataset (NSD), we adapt a generic human similarity judgment model (DreamSim) separately for each participant to obtain personalized perceptual similarity models. These models are then used to obtain subject-specific similarity structure across images in their encoding model training sets. We fine-tune each participant's baseline encoder in two conditions, aligning its representations with either generic or personalized similarity judgments, with the same initialization and objective. We find that baseline brain encoders reflect within-subject perceptual similarity more strongly than measured fMRI responses and preserve intersubject correlations present in measured fMRI, but fail to reflect those of human perceptual similarity judgments. Fine-tuning with generic, population-level perceptual similarity information results in better alignment with human judgments than baseline encoders, but these models still fail to capture intersubject correlations of human judgments. However, fine-tuning with personalized perceptual similarity information results in models that are the most aligned with human perceptual similarity judgments and also reflect intersubject correlations of human judgments. Our framework provides an approach for bootstrapping human similarity judgment data into more individualized, human-aligned brain encoding models of vision.