Ophthalmic vision language models benchmarked for image and spatial reasoning
EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Understanding eye images with AI is hard because eye diseases are complex and involve detailed features. The authors created EyeVQA, a large test set with questions about eye images that require recognizing diseases and pointing out exact locations. They tested many existing AI models and found they still struggle, especially when needing to look at multiple images or pinpoint locations. This work helps show where current AI falls short and where improvements are needed in eye image analysis.
What this means in practice
- •For medical imaging developers: Evaluate ophthalmic AI models' accuracy on complex questions involving disease detection and spatial localization using the unified EyeVQA benchmark.
- •For healthcare software teams: Improve AI-based eye disease diagnostic tools by identifying specific weaknesses in spatial understanding and multi-image reasoning within ophthalmic datasets.
Authors
Gujie Shao, Zixun Xie, Xuechun Xing, Ruixiang Wang, Ziyun Lan, Yanlin Qi, Gangyi Zhang, Yuxin Yang, Dawei Li, Haiming Tang
Abstract
Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insufficiently characterized. Existing ophthalmic datasets are typically designed for individual diseases or specialized tasks, making it difficult to systematically evaluate whether VLMs can move beyond disease recognition toward comparative reasoning and fine-grained spatial grounding. We introduce EyeVQA, a unified visual question answering benchmark for comprehensive evaluation of ophthalmic VLMs. EyeVQA is constructed from 21 available ophthalmic datasets and contains 20,000 question-answer pairs spanning six disease groups and seven question types: Single-Choice, Multi-Select, Variable-Select, True-False, Ranking, Point Location, and Bounding Box. Gold answers are deterministically derived from source-provided diagnoses, severity grades, clinical findings, segmentation masks, bounding boxes, and anatomical landmarks, enabling reproducible evaluation without relying on model-generated annotations. Notably, 44.5% of the questions require reasoning across multiple images, extending evaluation beyond conventional single-image medical VQA. We benchmark fourteen representative general-purpose, scientific, and medically specialized VLMs under a unified zero-shot protocol. The best-performing model only achieves an overall score of 62.8, while substantial gaps remain in spatial grounding and cross-task generalization. These results highlight the limitations of current VLMs in comprehensive ophthalmic visual understanding and establish EyeVQA as a diagnostic benchmark for developing more reliable and spatially grounded ophthalmic multimodal models. The project page is available at https://github.com/PKUTHM/EyeVQA.