Multilingual entity linking improves identification of rare items using reasoning and retrieval
Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking
Computation and Language
Summary
Finding the right place for names of things in pictures and text is tricky, especially when those things are not popular or well-known. The authors show that just looking at how often something is mentioned misses many rare items. They propose a method that uses smart searching and thinking over Wikipedia information to better connect these rare mentions to their real-world identities. This method works best when combining both searching and reasoning steps, improving results in multiple languages, especially for less common items.
What this means in practice
- •For multilingual ai developers: Improve entity recognition tools to correctly identify rare and less-documented entities across multiple languages and modalities.
- •For multimedia content managers: Enhance automated linking of text and images to accurate knowledge database entries when handling diverse and unusual entities.
Authors
Parinthapat Pengpun, Simran Khanuja, Graham Neubig
Abstract
Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4-39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.