FisherEdit improves knowledge updates in large language models
FisherEdit: Fisher Information Guided Knowledge Editing for Large Language Models
Computation and Language
Summary
Updating facts in large language models without breaking their other abilities is hard. The authors found that previous methods relied more on regularizing changes than on their designed projections. They propose FisherEdit, a new way to carefully change parts of the model that matter less while protecting important parts using Fisher information. This method keeps the model stable and accurate after updates, as shown in tests with several popular language models.
What this means in practice
- •For ai engineers: Enhance model updates by selectively protecting sensitive parameters to maintain overall performance during knowledge editing.
- •For natural language application developers: Improve reliability of language models when integrating new factual information without degrading downstream task accuracy.
Authors
Yizhou Yang, Jiahong Fu, Zhiwu Wang, Kailong Fan, Haokun Lin, Yichen Wu, Zhenan Sun, Renzhen Wang, Deyu Meng
Abstract
Sequential knowledge editing in Large Language Models (LLMs) often leads to a degradation of general capabilities due to cumulative parameter interference. In this paper, we revisit prominent projection-based methods like AlphaEdit and reveal a surprising mismatch: their robustness is primarily driven by update regularization rather than the intended null-space projection. Based on this insight, we propose FisherEdit, a theoretically grounded method built on Fisher geometry, which replaces coarse activation-space projections with importance-aware regularization in parameter space. It penalizes changes to sensitive parameters more strongly while retaining flexibility for new facts, and admits an efficient row-wise closed-form solution. Experiments on Llama-3, Qwen2.5, and Olmo-3 across multiple benchmarks demonstrate that FisherEdit achieves state-of-the-art performance while better preserving model stability and downstream task accuracy.