Humans and multimodal language models struggle similarly with finding objects in images
Human-Like Attention? A Psychophysical Comparison of Visual Search in Humans and MLLMs
Human-Computer InteractionArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Finding a specific object in a picture can be easy or hard for people, depending on how the objects look. This study shows that big AI systems that analyze both images and text, called multimodal large language models, make similar mistakes to humans when searching for things in pictures. Both humans and these AIs do well when the object stands out by color but have trouble when many features must be combined. However, humans take more time to be sure when the object is missing, while the AI tends to guess more strongly if the object is there or not.
What this means in practice
- •For computer vision engineers: Design better visual search algorithms by comparing AI errors with human difficulty patterns in 2D and 3D images.
- •For ai system testers: Improve evaluation of multimodal AI by using human-like difficulty signatures and error correlations in visual search tasks.
Authors
Renchi Zhang, Joost C. F. de Winter, Dimitra Dodou, Harleigh C. Seyffert, Yke Bauke Eisma
Abstract
Visual search is a fundamental cognitive ability. This study investigates whether Multimodal Large Language Models (MLLMs) exhibit human-like difficulty signatures in visual search tasks. We compared search performance of humans (n = 1,250) and MLLMs using identical 2D and 3D stimuli across different set sizes. Both groups showed efficient performance in feature searches, most clearly when the target had a unique color, but performance degradation in conjunction searches as set sizes increased. Additionally, we found strong correlations between human and MLLM error rates ($ρ= 0.82$), which suggests that MLLMs are sensitive to similar objective complexities, such as stimulus heterogeneity. However, differences were found as well: whereas humans invested extra search time to respond accurately on target-absent trials, MLLMs exhibited extreme present/absent response biases in complex searches. We conclude that MLLMs replicate high-level human performance signatures, yet their underlying computations differ significantly.