Explainability reveals how guardrails detect and miss prompt injections
Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection
Cryptography and SecurityArtificial Intelligence
Summary
Large language models can be tricked by sneaky inputs called prompt injections, so systems like Prompt Guard 2 try to detect these attacks. The authors used tools that highlight which words influence the detector’s decisions, finding it relies on many words combined rather than only a few. They also showed that changing some words moderately can fool the detector and let harmful prompts slip through. This means understanding these detectors can help improve security but also might help attackers find weak spots.
What this means in practice
- •For security engineers: Improve detection systems by analyzing which words influence prompt injection classifiers and harden weaknesses against synonym and paraphrase attacks.
- •For ai product teams: Design better safeguards for AI assistants by understanding failure modes of guardrails exposed through explainability techniques.
Authors
Fernando Outeda, Gustavo Betarte, Juan Diego Campo, Fiorella Cravero
Abstract
Large language models (LLMs) are increasingly deployed in production systems, raising concerns about their exposure to adversarial manipulation through prompt injection and jailbreak attacks. Classifier-based guardrails, such as Prompt Guard 2, are widely used as a first line of defense against such attacks, but their internal decision logic is largely opaque to both defenders and attackers. This paper presents an exploratory case study that applies explainable artificial intelligence (XAI) techniques to analyze how Prompt Guard 2 distinguishes malicious from benign prompts. We conduct four experiments to probe this question empirically. Guided by Vanilla Gradient and SHAP attributions, we find that Prompt Guard 2's decisions rely on the cumulative contribution of many tokens rather than a few dominant ones, yet saliency-guided synonym substitution and sentence-level paraphrasing can flip its predictions while altering only a moderate fraction of the text, in some cases yielding a successful jailbreak against the underlying LLM. A dataset-scale saliency analysis further shows that undetected injection prompts systematically lack the lexical markers the classifier relies on. We discuss the implications of these findings for the design and evaluation of classifier-based guardrails, and argue that explanation methods intended to support transparency can simultaneously lower the cost of constructing successful adversarial bypasses.