AI summaryⓘ
The authors studied how well computer models called pretrained vision embeddings match human brain responses when people look at urban street scenes. They found that models can predict how people rate scenes but do not organize visual information the same way human brains do. Even the best models explained less than 30% of the brain data patterns, and a simple Gabor energy descriptor did about as well as many complex models. They also showed that improving brain alignment in models makes them worse at predicting people's ratings, suggesting that good prediction of appraisals does not mean the model processes images like the brain does. Their evaluation uses public data without extra training, making it easy for others to test new models.
pretrained vision embeddingsurban scene appraisalEEGrepresentational geometryDINOv2Gabor energy descriptorhierarchical correspondencenoise ceilingself-supervised learningbrain-computer comparison
Abstract
Pretrained vision embeddings are increasingly used as general-purpose representations for modelling how people appraise urban scenes, and are validated almost entirely by how well they predict human ratings. High predictive accuracy does not establish that these embeddings organise scenes as human perception does. We test the two properties separately against brain data. Using openly released EEG from 63 adults who viewed and rated 56 Berlin street scenes, we estimate the representational geometry of the scenes over time, the proportion of that geometry that is explainable at all, and its correspondence with seventeen feature spaces spanning language-supervised, self-supervised, category-supervised and dense-prediction training, two orders of magnitude of scale, and interpretable controls. Correspondence is low throughout: the best representation, DINOv2 ViT-B, reaches 29.6% of the lower bound of the noise ceiling, the panel spans 11.0% to 29.6%, and a Gabor energy descriptor is indistinguishable from the best model while outperforming every language-supervised model tested. Within a model, deeper layers still match later neural responses, so the hierarchical correspondence found for object recognition survives even at this low overall level. The same embeddings predict held-out appraisal ratings well, up to r = 0.87, and the two measures do not track each other across models; reweighting features towards the neural geometry lowers appraisal prediction for every model tested, against a control of matched dimensionality. Predicting how a street is appraised is therefore weak evidence that a model represents the street as the brain does. The benchmark uses only public data and requires no training, so evaluating a new representation needs only its embeddings for 55 images.