Popular knowledge in language models spreads more errors when updated
Popular Knowledge Propagates More Errors in LLM Knowledge Updating
Computation and Language
Summary
Keeping language models up to date requires teaching them new facts, but updating can also cause them to forget or make mistakes. The authors studied which facts already known by the model are most likely to become wrong after updates. They found that facts linked to popular or well-connected topics are more easily corrupted, and errors about these topics can spread to related facts. To fix this, the authors created a method that keeps some popular facts locked in during updates, helping reduce mistakes.
language modelfine-tuningfactual updatinghallucinationforgettinglong-tail knowledgeknowledge graphrehearsal strategyentity popularity
Authors
Yuji Zhang, Weibing Wang, Cheng Qian, Duo Zhou, Dilek Hakkani-Tür, Kathleen McKeown, Chengxiang Zhai, Heng Ji
Abstract
Updating a language model's knowledge through fine-tuning is essential for keeping its outputs current, yet can also induce factual forgetting and new hallucinations. Prior work shows that long-tail knowledge is harder to acquire and newly memorized long-tail facts are difficult to retain during later fine-tuning. We study a complementary question: among facts that a model has encoded correctly, which are most vulnerable to collateral corruption during other updates? To investigate this question under a realistic factual distribution, we construct a large-scale graph FACTPROP of verified Wikipedia facts by linking triples that share head or tail entities, thereby preserving connections among factual knowledge. We fine-tune models on factual statements and measure correct-to-incorrect facts after each update. Our results reveal a pattern distinct from prior findings on long-tail vulnerability during acquisition and retention: among facts that models already answer correctly, those associated with highly connected entities are more likely to be corrupted by neighboring updates, and updates to such facts propagate errors more broadly. Structural popularity therefore predicts both vulnerability and downstream damage. Inspired by this finding, we propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.