Open-weight multimodal models struggle comparing unnamed body parts

Open-Weight MLLMs Cannot Compare What They Cannot Name

Computer Vision and Pattern Recognition

Summary

Multimodal large language models (MLLMs) can understand images and do complex visual tasks, but they have trouble comparing sizes of body parts they cannot recognize or name. The authors tested these models on simple images of ants and found that when the model cannot identify a body part, it fails to compare it properly, even if the comparison is visually easy. This suggests that these models depend on naming or recognizing objects before they can compare them. Their findings highlight a limitation where understanding what something is named as is needed for even basic visual comparisons.

What this means in practice

  • For biomedical imaging teams: Use findings to improve automated analysis tools that compare features in biological images by ensuring target structures are recognizable to AI models.
  • For ai application developers: Design vision-language systems for niche visual comparison tasks by accounting for the need of semantic recognition to enable accurate comparisons.

Authors

Wenxin Su, Anna Kreshuk

Abstract

Multimodal Large Language Models (MLLMs) are increasingly capable of understanding images and performing complex visual reasoning, suggesting the possibility of using them to automate biological image analysis. This need is becoming increasingly urgent as advances in technology continue to generate ever larger and more diverse microscopy datasets, already exceeding the scale at which manual visual analysis is feasible. With the final aim to use MLLMs for visual comparison of complex phenotypes, we investigate a very simple baseline: comparison of the size of body parts in synthetic images of ants. We find that state-of-the-art open-weight MLLMs fail for some body parts, but not for others, even though the comparison itself is visually trivial for all of them. We evaluate different in-context learning strategies and analyze performance on the related object recognition task. Our results suggest a recognition-driven grounding failure: structures that the models cannot recognize are instead associated with a recognizable semantic concept, impairing the grounding and comparison of both. Further analysis of the vision encoder is consistent with the hypothesis that fine-grained visual structure may be less well preserved without semantic support. Together, our results reveal a surprising limitation of current MLLMs: semantic recognition can gate simple visual comparisons that do not intrinsically require semantic understanding. Our code and data are available at https://github.com/kreshuklab/Open-Weight-MLLMs-Cannot-Compare-What-They-Cannot-Name