Prompt-injection detectors struggle with false alarms on tricky benign inputs
PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift
Cryptography and SecurityArtificial Intelligence
Summary
Detecting when someone tries to trick AI systems by injecting harmful instructions is important but tricky. The authors show that current detection methods often flag innocent, complex prompts as attacks, which wastes resources. They created a new benchmark to test detectors on different kinds of inputs, including tricky benign ones and disguised attacks. Their findings reveal that even top detectors make many false alarms on real-world benign data and that improving training on some hard examples doesn’t fully fix this problem.
What this means in practice
- •For ai safety engineers: Assess and improve prompt-injection detectors to reduce false alarms on complex benign inputs using a multi-axis benchmark.
- •For security operations teams: Better understand when prompt-injection detectors may wrongly block safe inputs, helping tune thresholds and training for more reliable alerts.
Authors
Yusuf Khalid Shire, Sang-Chul Kim
Abstract
Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under distribution shift, particularly on the benign side of the decision boundary, where false positives impose direct operational cost yet are seldom measured. We present PIDS-Bench, a frozen multi-axis benchmark that jointly evaluates attack detection and benign false-positive behavior at fixed thresholds, spanning in-distribution inputs, hard-benign prompts that mimic injection structure without malicious intent, obfuscated attacks, and domain and structural distribution shifts. We evaluate seven detectors (learned baselines, external prompt-injection classifiers, and broad-safety comparators) alongside a rule-based lower-bound reference. Multi-axis evaluation exposes a failure mode that aggregate F1 conceals. A detector exceeding F1 = 0.98 on the held-out split still misclassifies roughly one-third of an externally-sourced benign subset drawn from public corpora and restricted to security-adjacent content. Across a full threshold sweep and five training seeds, no internal detector reaches an operating point satisfying F1 >= 0.95 and hard-benign FPR <= 0.10 together on this stress distribution. Decomposing by provenance, we find that hard-negative augmentation nearly eliminates over-defense on curated stress inputs but leaves it substantially intact on externally-sourced prompts, a pattern we term provenance-sensitive over-defense. The asymmetry holds across both fine-tuned architectures and does not diminish as the augmentation pool grows, with the externally-sourced FPR remaining far above the 0.10 target. Whether augmentation matched to the externally-sourced distribution would close this gap is untested; threshold calibration and curated-style augmentation alone do not.