Large language model judges fail to match worker acceptance rates
Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement
Artificial IntelligenceComputation and LanguageComputers and SocietyMachine Learning
Summary
People often use large language models to judge if AI answers meet job standards. This paper shows that even when these models agree on which answers are better, they do not agree with how many answers real workers find acceptable or how occupations rate them overall. The authors tested many model setups and found that ranking answers well doesn’t mean the models estimate acceptance rates accurately. They suggest checking model scores against acceptance rates and group averages, not just rankings.
What this means in practice
- •For human resources teams: Improve AI evaluation tools by validating model scores against real worker acceptance rates and occupational aggregates to ensure workplace relevance.
- •For ai evaluation tool developers: Design AI judges that go beyond ranking to accurately estimate acceptance rates and occupational benchmarks for assessing workplace AI outputs.
Authors
Harry Lyu, Neil Thompson
Abstract
LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates. We introduce O*NET-BENCH, an audit suite derived from an existing survey of 45,796 worker ratings, and evaluate 33 pre-existing judge configurations across six model families on 4,501 test ratings. Twenty-five configurations achieve tie-aware pair accuracy of at least 0.60, although a train-fitted response-only TF-IDF baseline nearly matches the strongest judge. Despite this ordering agreement, judges estimate that 3.0%-97.9% of responses are acceptable, compared with 61.1% for occupation-matched workers. In one fine-tuned lineage, changing from pointwise scoring to a bundled few-shot/listwise protocol improves response ordering while reducing agreement with worker means at the task and occupation levels; this reversal replicates on a task- and worker-disjoint validation split under prespecified criteria. Cross-validated calibration largely removes mean bias, but calibrated scores explain at most 8.5% of individual worker-rating variance. Prediction-assisted estimation yields at most small precision gains at the studied label budgets. These results show that ranking agreement alone is insufficient for occupational measurement. Judges should be validated against the acceptance rates and aggregates their scores will be used to estimate.