Multimodal AI classifies Emirati homes architectural styles accurately
Multimodal Cultural Heritage Architectural Style Classification for Residential Buildings in the UAE Based on CLIP Embeddings and SVM
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Classifying architectural styles, especially for cultural heritage buildings, is hard because images alone can be confusing, and there is little data for non-western regions. The authors developed a computer method that looks at both photos of houses and expert-written descriptions together, turning them into a combined data format. They then grouped similar styles and trained a machine to recognize eight different Emirati residential architectural styles with 98% accuracy. This approach shows how combining pictures and text helps machines better understand complex visual categories.
What this means in practice
- •For cultural heritage organizations: Automate classification of regional architectural styles in heritage catalogs for better preservation management.
- •For urban planning teams: Identify and document architectural style patterns quickly in city planning by analyzing building images with expert descriptions.
Authors
Ahmed Ammar Kubba, Manar Abu Talib, Iman Ibrahim, Qassim Nasir
Abstract
The analysis and classification of cultural heritage architectural styles remain challenging due to the complexity of visual images of buildings, which are highly relied on in traditional CNN-based classification approaches in comparison to textual descriptions, and the relative lack of non-western region-specific datasets. This paper addresses this gap by proposing a multimodal machine learning framework to analyze and classify Emirati residential architecture using OpenAI's CLIP model. We integrate visual features from images and textual features from expert descriptions into a unified 512-dimensional embedding, followed by dimensionality reduction with UMAP for visualization and unsupervised clustering using K-Means. Cluster labels, which are derived from manual analysis of the K-Means clusters, are used to train an SVM classifier for automated architectural style classification. Our approach achieves a classification accuracy of 98% across eight identified style clusters, higher than every other study in the literature, demonstrating the effectiveness of combining visual and textual modalities. Overall, this paper highlights the potential of using multimodal AI to support architectural heritage analysis, offering scalable and interpretable tools for exploring regional architectural identities.