LLM judges show bias risks in evaluating entity alignment systems
Reliability of LLM Judges for Evaluating Entity Alignment
Artificial Intelligence
Summary
Entity alignment helps match things that mean the same across different knowledge databases, but checking if alignment systems work well needs expert human effort that is expensive and slow. The authors tested if large language models (LLMs) can replace humans as judges in this task and found that letting LLMs see the system’s answer can confuse them, causing them to rank systems incorrectly. By hiding system labels from the LLM judges, the authors recovered reliable evaluation results. They also released a new biomedical dataset and tools to help future evaluations of alignment systems.
What this means in practice
- •For knowledge base engineers: Use label-hidden LLM judges to scale evaluation of entity alignment without expensive expert annotation.
- •For biomedical data teams: Evaluate entity alignment between biomedical ontologies efficiently via validated LLM judge protocols and a new large benchmark dataset.
Authors
Vaibhava Lakshmi Ravideshik, Mayank Kejriwal
Abstract
Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.