Language models struggle to match names across different scripts accurately
Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models
Computation and LanguageArtificial Intelligence
Summary
Language models often decide that two names written in different scripts, like Devanagari and Latin, belong to different people even when they are the same person. The authors tested several models and found that accuracy drops sharply on same-person matches when names appear in non-Latin scripts. They identified a specific part of one model’s internal processing that causes this mistake and showed how adjusting it boosts accuracy significantly. The work shows that difficulties come not just from how names are written but also from how models make their final decisions.
What this means in practice
- •For multilingual ai developers: Improve name matching components to better recognize names in non-Latin scripts by adjusting model decision criteria and internal activations.
- •For international data integration teams: Enhance entity resolution workflows for datasets containing multiple scripts by diagnosing and correcting script-induced matching errors.
Authors
Athira Gopal, Patil Prathamesh Pravin, Ankush Kadu, Arpit Jain, Ashwanth Krishnan
Abstract
Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with attested Devanagari and Latin forms, holding the question and relation fixed. Qwen2.5-7B's same-person accuracy falls from 35.1% in Latin to 8.5% in Devanagari under Marathi prompts, while different-person accuracy remains above 98%. Evaluation across five models reveals distinct response biases, and reduced tokenization fragmentation does not reliably close the gap. Replicating the design on 374 audited Malayalam pairs built from the same entities reproduces the effect in a second script and language family: same-person accuracy falls to 5.3%, about half the matched Devanagari rate, while the margin still ranks matching above non-matching names at 0.925 AUROC. Latin-trained probes recover label information from Devanagari states, exceeding a string-feature baseline under similarity-matched negatives. Scalar calibration recovers 87% of the raw model-probe balanced-accuracy gap, identifying decision criterion placement as a major contributor. Activation patching localises causal sensitivity to the second entity in Qwen's middle layers; one shared sensitivity direction transfers about half the full-state margin correction. A fact-independent intervention at this site raises held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy. Together, these results distinguish script-dependent discrimination from criterion placement and identify a causal site for improving identity judgments.