Attack success rates vary widely depending on evaluation methods
Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates
Machine LearningArtificial Intelligence
Summary
People often test how well tricks to fool AI chatbots work by seeing if they get a certain response. This paper shows that depending on how you check answers, the success rates of these tricks can be very different. Sometimes repeating the test can reveal unsafe answers even if the AI said no at first. The authors suggest clearer testing rules and checking many responses to better understand AI safety.
What this means in practice
- •For ai safety engineers: Improve AI safety evaluations by using multiple evaluators and testing repeated interactions to detect hidden unsafe outputs.
- •For chatbot developers: Set design and testing standards for attack success that match response budgets and use validated evaluators to better ensure robust rejection of unsafe queries.
Authors
Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro, Eric Aubinais, Chih-Hong Cheng
Abstract
Jailbreak methods are commonly compared using Attack Success Rate (ASR), but these comparisons depend on how responses are generated and judged. In this paper, we examine eight attacks against three open-weight language models using three evaluation procedures. Low-resource language attacks achieve a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Changing the number and arrangement of attempts reverses the ranking of DeepInception and rewriting. In a separate experiment, sampling 100 responses for each unchanged request at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching but 50% under the Qwen3 evaluator. Beyond evaluator sensitivity, these findings raise a safety concern: repeated sampling can expose residual unsafe behavior despite an initial refusal, suggesting probabilistic suppression rather than robust exclusion. We recommend explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.