Summary
Matching things described in both pictures and words to the right real-world objects is hard, especially in art where clues can be subtle or missing. The authors created KARAMEL, a system that helps match artworks to entries in a knowledge base by combining image-text understanding with background knowledge about history and context. They use a special kind of neural network that looks at many different relationships between data to figure out which entity is the right one, even when the clues are indirect. They also built a new dataset called WikiMuSA, which connects artworks to known entities using images, text, and structured facts. Their experiments show KARAMEL works better than other methods, especially when the evidence is not clear-cut.
What this means in practice
- •For cultural heritage data teams: Use KARAMEL to automatically match artworks to known entities using images and texts, improving digital archive accuracy and search.
- •For multimodal data engineers: Employ KARAMEL's graph-based method to enhance entity linking in systems integrating visual and textual data with structured knowledge.
Authors
Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio, Giovanna Castellano, Ralf Krestel
Abstract
Discovering and disambiguating entities from multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior vision-language models that primarily rely on surface-level multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a vision-language encoder with a heterogeneous graph neural network. This design enables contextual reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based Multimodal Semantic data for Art), a new dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving multimodal observations and heterogeneous background knowledge. Both the dataset and source code are made publicly available.