Outdated documents cause ai models to give wrong answers often
Stale-Document Poisoning: When Outdated Retrieval Overrides Correct Model Answers
Computation and Language
Summary
When AI models look up information to answer questions, they sometimes find outdated documents that were true in the past but are now wrong. This outdated information can confuse the models and make them give incorrect answers, even if they would have been correct without looking things up. The authors tested many models across topics like medicine and law and found that outdated documents frequently cause errors. They also showed that telling models when old evidence stops being valid helps them switch to the right answer more reliably.
What this means in practice
- •For ai system builders: Design AI systems that detect and ignore outdated evidence to reduce incorrect model answers caused by stale documents during retrieval.
- •For legal tech developers: Incorporate temporal validity checks in AI tools for law to ensure answers match current standards and avoid errors from outdated legal documents.
Authors
Md Shamim Ahmed, Lukas Galke Poech, Richard Röttger
Abstract
Retrieval-augmented generation (RAG) is often used to address outdated knowledge by providing external evidence. But retrieval helps only when that evidence is still valid. We identify a temporal alignment failure, stale-document poisoning, in which outdated evidence makes a model wrong despite answering correctly without retrieval. We construct a benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy, grounded in dated official sources. Across 12 models, recent medical reversals are harder than long-established ones. More importantly, outdated retrieval flips 30% of Llama and 37% of Qwen answers even without instructions to trust the document; explicit follow instructions raise these rates to 66% and 75%. Across four open models and four domains, poisoning ranges from 17-91%, while matched up-to-date evidence is followed in 97-100% of trials. To isolate temporal applicability, we keep the historical evidence unchanged across 50 reversals and vary only the evaluation date. A clear pattern emerges: dates alone produce only modest adaptation, but when models are explicitly told when the old evidence stops applying, the larger models switch to the appropriate answer almost perfectly. Causal interventions confirm that this validity information directly shapes the final decision. The same internal components also support broader comparison tasks, suggesting that temporal applicability can recruit a general reasoning mechanism used for other comparisons. Finally, a fixed recency-aware hybrid re-ranker reduces poisoning by 4.6-10.0 points when dates are accurate, with gains that depend on reliable temporal metadata. Reliable RAG therefore requires selective trust: models must determine not only what retrieved evidence says, but whether it still applies.