Cost aware metric improves evaluation of iterative llm jailbreaks
Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks
Cryptography and SecurityComputation and Language
Summary
Evaluating how well tricks called jailbreaks work on large language models often misses that each try gives the model more chances to stop the attack. The authors introduce a new score that lowers the rating if the trick needs many repeated tries, highlighting how costly multiple attempts are. They tested this score on different jailbreak methods and found it changes the rankings, making some attacks look less effective when accounting for their cost. This helps clarify how expensive repeated attacks really are.
What this means in practice
- •For ai safety teams: Measure the true cost-effectiveness of iterative jailbreaks when testing model robustness to misuse attacks.
- •For security engineers: Prioritize defenses based on attacks that succeed efficiently without many model interactions, improving security resource allocation.
Authors
Rohan Menon, Chih-Hong Cheng
Abstract
Attack success rate (ASR) evaluates jailbreaks against large language models without accounting for repeated calls to the victim model. Yet each additional call gives the deployed system another opportunity to detect the attack and intervene. We introduce $\mathrm{ASR}_{\mathrm{STEP}}$, which penalizes additional victim-model calls while retaining binary success judgments. Our evaluation considers GCG~\cite{zou2023universal}, PAIR~\cite{chao2025jailbreaking} with a step budget, and a single-step PAP~\cite{zeng2024johnny} baseline on JailBreakBench across three model families. GCG's scores decrease by 7.0--8.7 percentage points after discounting, while PAP's remain unchanged. While for PAIR~\cite{chao2025jailbreaking}, we see a wider discounting range between 3.4-12 percentage points, showing that the "step" quantity differs per iterative strategy. Reporting both metrics makes the discount due to repeated victim-model interaction explicit. Code is available \hyperlink{https://github.com/SuELS-Group/beyond_asr}{here}.