Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Framework
2026-06-15 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors studied a system called Rapid Response that helps improve AI filters designed to catch harmful prompts, known as jailbreaks. They found that attackers can trick this system by injecting malicious examples into its training process, causing it to wrongly flag harmless prompts or miss harmful ones. Their new attack, called Omission Attack, works even when attackers can only alter harmful examples, making it harder to defend against. These attacks were highly effective even with very few poisoned samples, showing a serious risk to the safety filters.
Rapid Response frameworkjailbreak detectionprompt injectionpoisoning attacksbackdoor attacksfalse positivesfalse negativesmachine learning securitysynthetic training data
Authors
David Huang, Jaewon Chang, Avidan Shah, Prateek Mittal, Chawin Sitawarin
Abstract
The Rapid Response (RR) framework, deployed in production systems, including Anthropic's ASL-3 safeguards, continuously improves jailbreak-detection classifiers. When new jailbreaks emerge that bypass these classifiers, Rapid Response generates synthetic variants for training, helping the model generalize from the new attacks and quickly adapt. We reveal that prompt injection can infiltrate this pipeline to deliver poisoned samples into the classifier's training set, enabling two attack objectives: (I) targeted poisoning attacks that create false positives on harmless samples by categorizing them as a jailbreak, with a specific desired feature (e.g., certain formatting, subject, or keyword), (II) concept-based backdoor attacks that induce false negatives on jailbreak inputs, generalizing even to jailbreaks from attack strategies the defender explicitly trained against, when the backdoor trigger is present. Importantly, our threat model restricts adversaries to modifying only jailbreak samples (not benign data or labels), a constraint unexplored by prior work that makes the second objective particularly challenging. We address this with Omission Attack, which exploits a new phenomenon: when training on concept-absent unsafe samples, the classifier misassociates that concept's presence with the safe label. Both attacks cause substantial and in some cases near-complete label flipping at only a 1% poisoning rate, achieving up to 100% false positive rates and up to 96% false negative rates.