LLM graders vary widely in judging harmful response compliance
The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
Cryptography and SecurityComputation and LanguageMachine Learning
Summary
When people test language models for harmful or unsafe responses, they rely on 'graders' to judge if a response follows rules or breaks them. This paper finds that some graders unfairly lump all good responses into a rarely used category, making it look like the model breaks the rules more often than it really does. The authors show this problem happens across many models and graders, and suggest always reporting detailed counts of each response category to avoid confusion. This makes safety tests for language models clearer and more honest.
What this means in practice
- •For ai safety teams: Ensure more accurate evaluation of harmful content compliance by checking detailed grader label distributions during model safety testing.
- •For ai system auditors: Use per-class label counts alongside compliance rates to identify disparities in grader judgment and improve audit transparency.
Authors
Ruqiya Bin Safi
Abstract
Jailbreak papers report an attack success rate, the share of responses that deliver the harmful request, and read it as a property of the target model. We hold the target model, the prompts and the responses fixed and change only the grader. Three dense graders of 8B and 9B parameters, from three model families, move compliant responses to the neighbouring class, evasion, instead of scattering them across classes. One of them assigns the compliance label zero times in 900 responses. This class collapse also appears on new harmful requests, on a target model from a third family, and in a held-out 7B grader from a fourth family. Per-class label counts expose it at no cost and without human labels. The collapse decides how the rate behaves under a change of grader. On two corpora of 900 responses and eight graders, the compliance rate differs by a factor of 22 to 25, while the non-refusal rate differs by 18 to 23 percentage points. Without the three collapsing graders, compliance differs by a factor of 1.7 to 2.8 on all four corpora, and non-refusal still differs by 10 to 22 points. These results do not change when every grader is called once. We propose a reporting minimum that keeps the compliance rate and adds its per-class label counts and the named grader. Beside it goes the non-refusal rate, whose disagreement across graders is additive.