Toxicrag exposes risks of poisoning retrieval-augmented language models
ToxicRAG: Compromising Retrieval-Augmented Generation Systems via Single-Shot Knowledge Poisoning Attacks
Cryptography and Security
Summary
Some systems use external documents to help generate answers, but this approach can be tricked by fake information. This paper shows how an attacker can insert just one carefully crafted fake document that tells a persuasive story including false facts to change the system’s answer. The attack works by first agreeing with the old answer, then adding made-up events and citing fake experts to push a wrong answer. The authors tested this on various models and datasets and found the attack often succeeds, showing a new way misinformation can spread through these systems.
What this means in practice
- •For security teams: Evaluate and strengthen defenses against targeted misinformation attacks on retrieval-augmented AI systems.
- •For ai system developers: Improve factual consistency and source verification mechanisms in retrieval-augmented generation applications to reduce risks from poisoned knowledge.
Authors
Haozhe Lu, Jiaqi Li, Xinyuan Zhu, Xiang Li
Abstract
Retrieval-Augmented Generation (RAG) can ground large language model (LLM) outputs in external evidence, but it also exposes the system to knowledge poisoning. Representative attacks use multiple injected documents or templates that directly assert a target answer. We present ToxicRAG, a one-document-per-target attack that expresses misinformation as a coherent knowledge-update narrative. The generated document first acknowledges the previously accepted answer, introduces fabricated events that appear to invalidate it, and then attributes the attacker-selected answer to a set of purported authorities. An answer-focused self-validation loop optionally revises a candidate when a surrogate language model does not reproduce the target answer. We evaluate the attack on 100 target questions from each of Natural Questions, HotpotQA, and MS-MARCO, using four victim LLMs and four dense retrievers. In the sampled-corpus setting reported in this paper, ToxicRAG obtains ASRs between 0.61 and 0.91 across the twelve dataset--model combinations. It matches or exceeds the strongest evaluated baseline in every combination, with margins ranging from 0 to 11 percentage points. These results show that narrative-form poisoned documents can remain influential under the evaluated RAG configurations and motivate further study of factual consistency and source provenance in RAG systems.