Visual language models struggle to identify arabic calligraphy styles accurately
When Models Misread Ink: Auditing VLMs for Arabic Calligraphy Recognition
Computer Vision and Pattern Recognition
Summary
Arabic calligraphy is an important cultural art form with many detailed styles. The authors tested 12 visual language models (VLMs) on recognizing six different calligraphy styles but found that the models often guessed one style too frequently, missing many others. Some ways of asking the models helped a little but didn’t solve the problem. This shows that recognizing calligraphy is a tough challenge for models and helps reveal which visual styles AI can understand better.
What this means in practice
- •For digital heritage preservation teams: Assess the reliability of existing AI models for automatically tagging and cataloging Arabic calligraphy artworks in cultural archives.
- •For multilingual ai developers: Improve visual language models to better differentiate detailed Arabic script styles to enhance text recognition in diverse language datasets.
Authors
Niaz Mohaiman Abtahi, Evan Maksud, Reyana Shahid
Abstract
Arabic calligraphy is a high-stakes cultural visual domain for Muslim communities, yet VLM evaluations rarely test its fine-grained script traditions. We audit 12 VLMs on a cleaned 1,195-image six-way benchmark using numeric, visual-cue, and Diwani-first prompts. In this closed-set setting, the best numeric-prompt accuracy is 60.9% for closed/API models and 47.2% for open-weight models. The main failure is default-label collapse: every numeric run is dominated by a single label taking 33-63% of predictions, against an 18.8% majority-class baseline. For 8 of 12 models the default is Thuluth; GPT-4o labels 52.9% of all images as Thuluth although it is only 18.1% of the set. Prompting helps unevenly: visual cues move accuracy by more than 10 points for some models but hurt others, and foregrounding Diwani often relocates the default rather than removing it. Arabic calligraphy recognition is therefore a compact, culturally grounded stress test for VLM robustness and for measuring which visual traditions become legible to automated systems.