Addressed memory updates improve targeted knowledge injection in language models
Writability and the Pathway: A Controlled Study of Knowledge Injection into Addressed Memory
Computation and Language
Summary
The paper studies how to add new knowledge to a language model’s memory after it is trained, so it answers new questions correctly without messing up old answers. The authors tested different methods of updating parts of the model and found some ways to improve recall for specific target answers, but the updates didn't guarantee full correct answers or perfect behavior. They showed it is possible to make controlled changes to memory in models like GPT-2 and larger Qwen and DeepSeek models, but the process is limited and complex. This work helps understand how injected knowledge behaves in model memory and what challenges remain.
What this means in practice
- •For language model engineers: Improve post-training updates to language models to inject new facts without fully retraining or losing old knowledge.
- •For software maintaining ai systems: Develop procedures to control knowledge updates in deployed language models while minimizing undesired changes in behavior.
Authors
Wei Gao, Cheng Tian, Duan Yang, Jianqing Yin
Abstract
Addressed conditional memory provides a local surface for post-hoc knowledge injection, but a successful write need not transfer to a new question or preserve untargeted behavior. This study combines a controlled GPT-2 comparison with production-memory experiments on Qwen3.8-Flash-Next-FP8 and DeepSeek-V4.1-Flash. Newly initialized modules attached to frozen GPT-2 are tested in a 2 x 2 design: row-only versus row-plus-projection updates, each with or without fixed module-residual anchoring. All arms reproduce 50 trained completions in each of three sampling seeds. Natural-question target-string hits average 5.3% for both row-only arms, 28.7% for unanchored projection updates, and 22.0% with anchoring, while strict answer scores remain zero. Anchoring lowers measured drift; lexical hits do not establish entity-specific retrieval. In the archived Qwen experiments, gradient writes make the designated first answer token top-ranked for 132 independently reset targets, comprising 93 new promotions and 39 already top-ranked targets. A separate matched 24-target comparison gives 13 new promotions with Adam versus two with SPSA at equal iteration counts but unequal computational procedures. An 80-row case includes saved continuations and a replay record, while reformulations bound its scope. Six DeepSeek zero-order cases yield one new target-token promotion and one regression. These results support bounded writing feasibility and separate trained expression, question-form behavior, and preservation; first-token rankings do not establish complete-answer success or universal conditions for writable memory.