Historical manuscript illustrations classified with fine-tuned vision models
A Visual Classification Dataset and Model Evaluation for Historical Manuscript Illustrations
Computer Vision and Pattern RecognitionMachine Learning
Summary
Old manuscripts contain many pictures showing people, animals, plants, and more, but it's hard to sort and search these images online. The authors created a new dataset with 15,000 labeled historical illustrations across 22 groups and tested different computer vision methods to automatically classify these images. They found that models specially trained on this task work much better than general-purpose ones. Their work also shows that some image categories are hard to separate because the pictures overlap in meaning.
What this means in practice
- •For digital archivists: Automate sorting and searching of large manuscript illustration collections using fine-tuned image classifiers.
- •For cultural heritage platform developers: Integrate visual classification to improve user browsing and exploration of historical image archives online.
Authors
Yoav Evron, Michal Bar-Asher Siegal, Michael Fire
Abstract
Historical manuscript illustrations preserve rich visual evidence of past cultures. They depict people, animals, plants, diagrams, music notations, and decorative forms. Although large digitization projects have made many manuscripts available online, the material itself remains difficult to explore at scale. Extraction systems can find illustrations on manuscript pages, but without meaningful categories, large collections remain hard to search and explore. We address this gap by introducing a manually labeled dataset of 15,000 illustrations from manuscripts dating back hundreds of years across 22 categories, and evaluating modern vision models for image classification on this task. The problem is challenging due to stylistic diversity, degradation, and semantic ambiguity, with many images that fit more than one category. We compare fine-tuned CNN and Transformer-based classifiers, zero-shot CLIP, embedding-based classifiers, and direct vision-language models. Results show that fine-tuned image classifiers perform best overall, with ConvNeXt reaching 88.9% accuracy and 81.3% macro-F1. Using CLIP embeddings with XGBoost provides a strong alternative. In contrast, zero-shot CLIP and direct vision-language classification perform substantially worse, highlighting the limits of general-purpose models in this domain. Beyond overall performance, the analysis reveals which categories are visually separable and where errors reflect genuine semantic overlap, suggesting that some limitations arise from the taxonomy itself.