Guardrail models flip labels when prompts are repeated many times
Overflip: Repetition-Induced Label Flips in Guardrail Models
Artificial Intelligence
Summary
Some systems that check if input to AI language services is safe can change their judgment when the same prompt is repeated over and over. The authors call this problem Overflip, where a harmful prompt might suddenly be marked as safe after many repetitions, even though its meaning doesn't change. This happens because the model's attention spreads equally over the repeated parts, causing unpredictable flips in decisions. This finding reveals a weakness in current guardrail checks and suggests the need for more robust methods.
What this means in practice
- •For ai safety engineers: Detect Overflip vulnerabilities in guardrail models to improve harmful content filtering under long inputs.
- •For cloud service providers: Enhance evaluation protocols of guardrail models to prevent malicious prompt bypass in AI service offerings.
Authors
Xu He, Chih-Hsuan Lin, Hung-Mao Chen, Junjie Xiong, Yan Zhai, Kun Sun
Abstract
Guardrail models are classifiers deployed to screen malicious prompts and responses in LLM-based services. To meet latency constraints, many lightweight guardrails adopt compact Transformer backbones (e.g., DeBERTa) that are trained with short context windows (typically 512 tokens) and rely on bucketed relative positional encodings to process longer inputs. Prior evaluations assume that a guardrail's decision is stable as the input is lengthened. We show that this assumption can fail. We identify Overflip, a repetition-induced instability where repeating a prompt causes the guardrail's prediction to flip (MAL$\to$BEN) as the sequence grows. We conduct experiments on 9 widely used lightweight guardrail models. Five exhibit MAL$\to$BEN flips on a benchmark of 100 prompts, with confidence margins shrinking steadily with repetition. Among these vulnerable models, flip rates range from 8% to 92%, with first flips occurring at roughly 2.6k--9.4k tokens. Our analysis suggests Overflip differs from traditional attention-dilution baselines, which aim to divert the model's attention away from tokens associated with malicious content, shifting it instead toward unrelated content, such as benign padding or shuffling. While Overflip preserves malicious content, it homogenizes token-level attention over repeated structure and induces a distinct, more gradual attention-dispersion trajectory than padding. Moreover, Overflip poses a greater threat to LLM services than traditional attention dilution methods. Because the bypassed prompt remains semantically intact and is still readily understood by downstream business LLMs, it can transmit malicious intent after passing the guardrail. These findings expose repetition as an attack surface for guardrail models and motivate length-robust evaluation and mitigation.