Parametric memory influences large language models reasoning performance
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs
Computation and Language
Summary
Large language models (LLMs) do well on reasoning tests, but it was unclear if they actually think through problems or just remember facts they learned before. The authors created a test called MemoReason, which swaps real facts for made-up ones while keeping the questions the same. They found that the models perform worse with made-up facts, showing they rely on memory, but they don’t just spit out memorized answers without reasoning. This means the models’ memory affects reasoning in more complex ways than just recalling facts.
What this means in practice
- •For llm developers: Improve large language model designs by understanding how internal memory affects reasoning accuracy beyond simple fact recall.
- •For ai testing teams: Use MemoReason benchmark to evaluate and diagnose reasoning failures in language models when dealing with unfamiliar or fictitious information.
Authors
Zineddine Tighidet, Andrea Mogini, Jiali Mei, Patrick Gallinari, Benjamin Piwowarski
Abstract
Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves reasoning performance, and the \textit{Strong Parametric Shortcut Hypothesis}, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce \textbf{MemoReason}, a human-curated benchmark that pairs factual reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7\% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. \textbf{MemoReason} provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader reasoning settings.