Dynamic editing improves large language model updates on tricky information
DynEdit: Dynamic Entropy-Guided Sequential Editing for Large Language Models
Computation and LanguageArtificial Intelligence
Summary
Large language models sometimes hold outdated or wrong information that needs fixing without retraining the entire model. The authors find that certain parts of the text, those with higher uncertainty or 'entropy,' are harder to update but important. They create a new method called DynEdit that focuses editing efforts more on these tricky, key parts instead of treating all parts equally. This approach leads to better updates over long texts, making the knowledge corrections more accurate and effective.
What this means in practice
- •For ai platform engineers: Implement targeted updates to large language model knowledge in production without costly full retraining, improving update accuracy on complex text regions.
- •For automated content management teams: Correct or update factual content in long-form generated text by focusing on key information-dense segments identified through entropy.
Authors
Jinhu Fu, Yan Bai, Yihang Lou, Li Sun, Jiahong Wu, Xiangxiang Chu, Sen Su
Abstract
Knowledge editing aims to update outdated or incorrect knowledge in large language models without full retraining. Existing parameter-editing methods remain limited for long-sequence knowledge: single-point editors provide insufficient coverage for extended targets, while fixed chunk-based editors distribute editing effort uniformly across rigid predefined windows, failing to prioritize difficult, information-dense tokens. In this work, we show that token entropy is closely associated with both editing difficulty and semantic importance: high-entropy regions are harder to edit and often contain key factual content. Motivated by this observation, we propose DynEdit, a dynamic framework for long-sequence knowledge editing. DynEdit uses token-level entropy to guide non-uniform sliding, increasing the coverage of difficult high-entropy regions, and further applies entropy-guided token-level optimization to strengthen updates on information-dense tokens within each editing window. Experiments on three benchmarks and two backbone models show that DynEdit consistently outperforms competitive baselines across diverse long-form editing settings, achieving gains of up to +14.0 BLEU points on challenging paraphrase queries and near-saturated semantic scores on several diverse knowledge domains.