Diverse distributed attacks improve poisoning of retrieval augmented generation
Divide and Doubt: Diverse Distributed Poisoning for Retrieval-Augmented Generation
Cryptography and Security
Summary
When language models answer questions by looking up information from multiple documents, attackers can poison the databases with false claims. Previous methods repeated the same fake claim in similar documents, which some defenses can spot and block. The authors propose a new attack called DnD that spreads the fake claim across many different styles of documents and includes one document casting doubt on the real answer, making the attack harder to detect. They tested this attack on various question-answer setups and found it often works better, especially against defenses that look for similar or conflicting information.
What this means in practice
- •For security teams: Evaluate and improve defenses for systems combining language models with document search against advanced poisoning attacks like DnD.
- •For enterprise ai developers: Test robustness of question-answering AI systems that retrieve documents to ensure reliability against diverse poisoning strategies.
Authors
Tianhao Chen, Yuhan Wei, Weifei Jin, Zhengyuan Jiang, Yuepeng Hu, Neil Zhenqiang Gong
Abstract
Multi-passage corpus poisoning often repeats one target claim across similar documents, creating correlated lexical and semantic patterns that similarity- and conflict-aware defenses can suppress jointly. We introduce DnD (Divide and Doubt), a targeted attack based on two principles: distributing support for the target answer across stylistically diverse passages, and including a passage that casts doubt on evidence for the reference answer. The first disperses poison-passage representations in embedding space, while the second strengthens target adoption when multiple poisoned passages are retrieved. We evaluate DnD on two open-domain QA datasets across three LLMs and nine RAG configurations, under both black-box and white-box access to the retriever. Across these settings, DnD matches or outperforms prior attacks in most configurations, with its largest gains against clustering- and conflict-aware defenses.