Language models balance memory and context in factual recall differently
Memory vs. Context? Influential Factors of Factual Recall in Language Models
Computation and LanguageArtificial Intelligence
Summary
This paper looks at how language models decide between using facts they have memorized and new information presented in the text they are reading. The authors repeated earlier experiments and tested many different models to see how well previous findings hold up. They found that bigger models usually rely more on what they remember, but this varies by model type and other factors. Sometimes the way a question is asked can change whether a model trusts its memory or the new context a lot. They also found that even unrelated text can sometimes fool the model into trusting the context.
What this means in practice
- •For chatbot developers: Design chatbots that better balance predefined knowledge with new in-conversation facts by tuning model size and prompt phrasing.
- •For ai content moderators: Improve detection of when language models might be misled by irrelevant or contradictory context in user inputs.
Authors
Guilhem Fouilhé, Nicholas Asher, Philippe Muller
Abstract
We reproduce and stress-test the work of Yu et al. (2023), who characterize how language models (LMs) arbitrate between memorized knowledge and contradictory in-context statements. We replicate their world-capitals experiments on 31 models spanning Pythia, GPT-2, Qwen3, and Ministral families, including base and post-trained variants, and extend evaluations to five additional knowledge relation types from the ParaConflict dataset. We empirically confirm most of their original findings: larger models and higher-frequency entities tend to favor memorized answers, with substantial family-level variance. However, several conclusions do not generalize cleanly: entity-frequency effects disappear on Qwen3-14B and 32B; post-training shifts the memory-context trade-off inconsistently across families; question phrasing alone can change a model's reliance on memorized knowledge by up to 80 percentage points; and semantically unrelated prose can mimic coherent supporting context. Our results clarify where Yu et al.'s claims hold and to what extent they generalize to other prompts.