Language models struggle to update memories based on time changes
Engram Injection: A Study of Temporal Validity in Parametric Language-Model Memory
Artificial Intelligence
Summary
Sometimes facts that a computer knows stop being true because things change over time. The authors tested if a language model can learn to remember facts only when they are valid, based on the current time. They tried storing facts in parts called adapters and giving the model a clock input, but it couldn’t correctly answer questions about facts changing after a week. Other methods that show the model all past updates or retrieve the right record worked much better. This shows it’s harder for language models to learn when to use a fact, not just to learn the fact itself.
What this means in practice
- •For chatbot developers: Improve chatbot responses by integrating external retrieval systems for up-to-date facts instead of relying solely on learned model memory.
- •For knowledge management teams: Manage and query evolving facts by combining stored knowledge with timestamped retrieval for accurate change tracking.
Authors
Sam Siavoshian, Omar Ramadan, Amir K. Saeed, Benjamin A. Johnson, Benjamin M. Rodriguez
Abstract
A stored fact may stop being valid when a user changes a preference or an event replaces an earlier one. We study whether a language model can learn these changes in its parameters and answer correctly from only a question and the current time. Engram Injection stores facts in LoRA adapters and supplies the clock through FiLM conditioning on a frozen Qwen2.5-3B-Instruct model. We compare it with additive clock conditioning, prompting with the full dated history, and retrieving four dated records. We train six matched adapters across three seeds after fixing a sampler that could admit test-age examples into training. The training audit reports no such examples among 127,760 evaluated training inputs. On 62 pairs of questions asked before and after an update, at an elapsed age of seven days excluded from training, both adapter methods answer zero pairs correctly under the primary blinded assistant review with order code 1 held fixed. Full-history prompting answers 59/62 pairs correctly, and retrieval answers 62/62. Separate experiments show that adapters can store facts and that removable memory units can support local deletion. Combining fact selection with time-based version selection remains unsuccessful. These results separate learning a fact from learning when to use it.