Context changes which memorized examples large language models reveal
Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes
Artificial Intelligence
Summary
Large language models can repeat pieces of text they saw during training when given the start of that text. This study looks at whether adding extra information around that start, like instructions or related documents, stops the model from repeating those memorized parts. The authors find that sometimes the extra context hides some memorized texts, but other times it reveals new ones. So, adding context changes but does not remove the risk that models expose memorized data.
What this means in practice
- •For model deployment engineers: Improve risk assessments of data leakage by testing model outputs under varied context conditions, not just isolated prefixes.
- •For security analysts: Detect and monitor sensitive memorized outputs by considering how retrieval or prompt context may enable new data exposures.
Authors
Ali Satvaty, Narjes Sharafi, Jirui Qi, Suzan Verberne, Fatih Turkmen
Abstract
Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation. In deployed systems, however, prefixes are rarely evaluated in isolation. They often appear together with instructions, retrieved documents, or other task-specific context, as in retrieval-augmented generation (RAG). This motivates examining whether contextual conditioning mitigates memorization or merely changes the set of memorized samples that become extractable. We investigate this issue through paired item-level measurements of probabilistic suffix extraction. For each prefix-suffix pair, we score the target suffix under an empty prompt and under retrieved contexts of varying relevance, across three open-weight instruction-tuned models. We find that context does not simply erase memorization. Instead, extractable memorization consists of a context-robust core and a context-sensitive boundary. Many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases. At the same time, context mainly affects marginal samples near the extraction threshold: it suppresses some exposures, but also enables new ones that are missed by prefix-only evaluation. These findings qualify the view that RAG reduces memorization risk. Context can lower aggregate extraction by suppressing boundary cases, yet robustly extractable samples persist, and context-enabled extractability remains security-relevant.