Large language models edited with geometry aware knowledge protection

Editing Large Language Models with Geometry-Aware Regularization

Computation and LanguageArtificial Intelligence

Summary

Updating the knowledge inside large language models can be tricky because some facts are more important to keep, and some updates are harder than others. The authors introduce GEdit, a new way to edit these models that respects the underlying math structure of how knowledge is stored. This method protects the most important knowledge better and adjusts how strictly it edits based on how difficult the update is. Tests on popular large language models show GEdit works better than previous methods by both keeping old knowledge intact and successfully making new edits.

What this means in practice

  • For ai model engineers: Update large language models with targeted protection of core knowledge while handling many edits safely and effectively.
  • For customer support automation teams: Continuously improve language model responses by sequentially updating model knowledge without degrading existing capabilities.

Authors

Bingqing Liu, Wei Liu, Jun Wang, Xiaobo Sun, Yuhua Li, Ruixuan Li

Abstract

Knowledge editing has become a key technique for updating knowledge in large language models. However, existing methods neglect the geometric structure of the knowledge representation space, leading to two key limitations: they cannot distinguish which pre-trained knowledge should be preferentially preserved (core vs. non-core) nor which new edits are inherently difficult (hard vs. easy). To address this, we propose GEdit, a geometry-aware framework built on a Low-Rank Plus Diagonal (LR+D) factor model that captures the intrinsic low-rank structure of Transformer FFN hidden representations. GEdit features two complementary mechanisms. First, the low-rank regularization applies targeted protection on a low-rank core knowledge subspace. Second, the adaptive regularization dynamically adjusts the regularization strength based on each edit request's editability: relaxing constraints for hard edits to ensure success, and tightening them for easy edits to better preserve pretrained knowledge. Extensive experiments on GPT2-XL, GPT-J, and LLaMA-3-8B under large-scale editing scenarios, including both mini-batch and fully sequential editing settings with up to 10,000 continuous updates, show that GEdit consistently and significantly outperforms state-of-the-art baselines in both editing success and knowledge preservation.