Improving fact recall in language models with focused context views
Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing
Computation and LanguageArtificial Intelligence
Summary
Large language models know lots of facts but can forget or mix up details that change after they are trained. The authors found that when these models are updated using paragraphs that contain many facts, they tend to remember the whole paragraph but struggle to recall specific facts on their own. To fix this, the authors created a method called FOVEATED that tricks the model into paying attention to each fact separately by changing the order of words around it during training. This helps the model better remember individual facts without needing the original paragraph nearby.
What this means in practice
- •For language model engineers: Improve updating specific facts in LLMs without retraining the whole model using focused context perturbations.
- •For knowledge management teams: Enhance reliability of fact updates in AI systems that rely on free-form text edits rather than structured data.
- •For search engine developers: Support more accurate fact updates inside language-based search features without losing context.$Commercial implications: Enables selling improved factual accuracy in AI-powered search by updating knowledge more efficiently and reliably.
Authors
Ding Wu, Ye Zhang, Haoyu Wang, Tianci Liu
Abstract
Large language models (LLMs) increasingly serve as general-purpose interfaces to factual knowledge, but their parameters do not automatically reflect information that changes after pretraining. Knowledge editing (KE) provides a targeted alternative to costly retraining by modifying selected knowledge and preserving unrelated knowledge and general capabilities. Conventional KE uses structured factual triples, whereas unstructured KE (UKE) uses free-form passages containing multiple facts. Nonetheless, existing UKE editors exhibit a failure mode known as context reliance: edited LLMs can often reproduce the editing passage but fail to reliably recall its individual facts without the original passage context. We identify context-induced difficulty underestimation under the standard passage-level editing objective: later facts receive increasingly rich ground-truth context and consequently incur lower initial losses, making them appear easier to learn. In response, we propose FOVEATED, a plug-and-play framework that constructs focused views of each sentence by randomly shifting the Rotary Position Embedding (RoPE) positions assigned to the keys of its preceding context. The perturbation is applied during editing and removed afterward, leaving the model's native positional encoding unchanged at inference time. We instantiate FOVEATED for both direct-optimization and locate-then-edit editors. We theoretically analyze how FOVEATED counteracts context-induced difficulty underestimation and empirically demonstrate consistent improvements across five KE editors, two LLM backbones, and three benchmarks.