ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
2026-08-31 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors organized a challenge called ImageEval 2026 to test how well computer systems understand images and language in Arabic and English. The challenge had two parts: one focused on answering questions about images using spoken language and spotting mistakes in image descriptions, and the other on checking if AI-generated images match cultural details. Fourteen teams entered, using different methods like specialized models and combining results. The authors explain the tasks, data used, how they judged the systems, and share all the resources so others can use them. They found that understanding culture through images and Arabic speech is still quite difficult for AI.
Visual Question AnsweringMultimodal EvaluationArabic Language ProcessingSpeech RecognitionText-to-Image GenerationCultural AccuracyVision-Language ModelsZero-Shot LearningFine-TuningEnsembling
Authors
Samir Abdaljalil, Hunzalah Hassan Bhatti, Ahlam Bashiti, Farina Amir, Md Arid Hasan, Basel Mousi, Nadir Durrani, Fahim Dalvi, Zien Sheikh Ali, Erchin Serpedin, Hasan Kurban, Mustafa Jarrar, Shammur Absar Chowdhury, Firoj Alam
Abstract
We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation. It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation. A total of 14 teams participated in the test phase, with 12 teams submitting system description papers. Participating systems used a range of approaches, including zero-shot prompting, fine-tuning of vision-language models, speech-recognition pipelines, ensembling, and score calibration. We describe the task setup, datasets, evaluation procedure, and participating systems, and summarize the main results across the different tracks. All datasets and evaluation scripts from the shared task are released to the research community. The shared task highlights the challenges of culturally grounded multimodal evaluation, particularly for Arabic speech and image-text reasoning.