Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms
2026-04-23 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models remember facts about things when those things are called by different names. They made a new test called RedirectQA that uses different spellings and nicknames from Wikipedia to see if models still get the facts right. Their results showed that models can be confused when the name of an entity changes, especially with bigger changes like using abbreviations instead of minor spelling differences. This means models don't just memorize facts attached to one exact name but aren't completely flexible either. The authors suggest it's important to test language models using many different names for the same fact to better understand their knowledge.
large language modelsfactual memorizationentity-based question answeringWikipedia redirectssurface formsaliasesorthographic variationWikidatanon-verbatim memorization
Authors
Yuto Nishida, Naoki Shikoda, Yosuke Kishinami, Ryo Fujii, Makoto Morishita, Hidetaka Kamigaito, Taro Watanabe
Abstract
Understanding what kinds of factual knowledge large language models (LLMs) memorize is essential for evaluating their reliability and limitations. Entity-based QA is a common framework for analyzing non-verbatim memorization, but typical evaluations query each entity using a single canonical surface form, making it difficult to disentangle fact memorization from access through a particular name. We introduce RedirectQA, an entity-based QA dataset that uses Wikipedia redirect information to associate Wikidata factual triples with categorized surface forms for each entity, including alternative names, abbreviations, spelling variants, and common erroneous forms. Across 13 LLMs, we examine surface-conditioned factual memorization and find that prediction outcomes often change when only the entity surface form changes. This inconsistency is category-dependent: models are more robust to minor orthographic variations than to larger lexical variations such as aliases and abbreviations. Frequency analyses further suggest that both entity- and surface-level frequencies are associated with accuracy, and that entity frequency often contributes beyond surface frequency. Overall, factual memorization appears neither purely surface-specific nor fully surface-invariant, highlighting the importance of surface-form diversity in evaluating non-verbatim memorization.