AI summaryⓘ
The authors studied how multimodal large language models (MLLMs) perform visual search compared to humans by simulating eye movements on the same images. They found that while the models often detected targets as well as or better than humans, their way of 'looking' at the image was different—models made very consistent, big jumps without the step-by-step scanning humans do. This suggests that even with the same visual input, models don't mimic the timing and process of human vision, only the final locations looked at. The authors conclude that common evaluation metrics miss these differences, so models might answer 'what' or 'where' questions well, but not 'how' the visual search happens over time.
visual searchfoveascanpathmultimodal large language modelsCOCO-Search18fixationsaccadeattention alignmententropysaliency metrics
Authors
Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Ulas Bagci, Alessandro Bruno
Abstract
Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath. Whether multimodal large language models (MLLMs), given the same foveated input, search as humans do bears on their use as models of human vision and on attention-alignment scores. We compare three general-purpose MLLMs with human eye-movement scanpaths on goal-directed search (COCO-Search18), driving each model fixation by fixation through an identical, human-matched foveated view and assessing it along three axes: the decision of target presence, the efficiency of reaching the target, and the gaze process itself. The axes dissociate. On the decision and on target acquisition the models match or exceed humans, detecting present targets near ceiling and reaching them on the first saccade more often than people do. The gaze process is not human. Under the human-matched condition, all three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other. That is consistent with a single-pass, non-serial architecture rather than a limit of acuity. Matched retinal input reproduces where humans look but not how the looking unfolds in time, and no degradation regime recovers human-like search at human-like success. The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level ones.