AI summaryⓘ
The authors study how well safety scores predict harmful prompts before text is generated by language models. They find that these scores sometimes misjudge prompts, especially when a prompt is wrapped with extra content, causing harmful outputs to increase while the scores incorrectly say the prompts are safer. This mismatch means that the scores fail to catch many harmful attacks that actually succeed. The authors introduce a method called Active Attention Probing to better measure attention signals and show that the problem is consistent across different models, attack types, and evaluation methods. Overall, the study highlights how changes in input can trick safety scores and reduce their reliability.
internal safety scoresharmful promptsjailbreak attacksattention mechanismsActive Attention ProbingAUROCdistribution shiftlanguage modelsprompt wrappingmodel evaluation
Authors
Mingyu Luo, Ming Deng, Zilang Qiu, Yiming Cheng, Ci Tao, Xue Tan, Sijin Sun, Yangfu Li, Ping Chen, Jun Dai, Xiaoyan Sun
Abstract
Internal safety scores judge a prompt before any text is generated, and they are validated by how well they separate harmful prompts from benign ones. That separation is then read as evidence that the score will also catch the attacks that succeed. Harmful intent is a property of the prompt. Jailbreak success is an outcome produced later by a particular target model, decoding policy, and judge. A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway. In this paper we audit that inference. Attention based measurements are usually read from prompt dependent locations, so a wrapper changes both the content being judged and the place the signal is taken from. We therefore introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate. We pair every base goal with a plain and a wrapped version and generate real completions from the target models. On Llama, wrapping raises harmful generation from 0.05 to 0.27 while harmful intent AUROC falls from 0.936 to 0.803, so the attacks grow more dangerous while the prompts look safer to the score. Among wrapped harmful prompts the outcome AUROC is 0.220, which places the attacks that succeeded below the attacks that failed. Rare token, passive, and detector derived channels reproduce the reversal on the same matched design, and the reversal itself persists across three target models, seven attack families, and two independent judges. Distribution shift then degrades calibration and threshold transfer before it degrades ranking.