Open contrastive decision model performs near chance on judge benchmarks
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Machine LearningArtificial IntelligenceComputation and Language
Summary
This paper studies how well a type of AI model called an open contrastive decision model works as a judge on difficult tests. The authors found that this model performs about as well as random guessing on several public benchmarks. Other similar models with the same size perform much better at judging. The paper also shows that while the model’s confidence scores are too high, calibrating them can improve how well confidence indicates errors. However, using these confidence scores to decide when to rely on stronger judges doesn’t work well because the model rarely has confident enough answers.
What this means in practice
- •For ai model evaluators: Use benchmark results to compare reliability of decision models and identify calibration methods for confidence scores.
- •For quality assurance teams: Improve model selection criteria by understanding limitations of contrastive decision models as judges on preference datasets.
Authors
Gowthamkumar Nandakishore
Abstract
An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item with one constant label, matching the trivial always-first baseline at 0.581. Judges with the same parameter count score far higher everywhere: a reward model reaches 0.764 to 0.976 and a generative judge 0.611 to 0.778, and every gap to CLM is significant after Benjamini-Hochberg correction. Two properties do work. Raw confidences are overconfident by up to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062, and the repaired confidence ranks the model's own errors above chance on three of six benchmarks. The decision order-flip rate is 0.0002 against 0.2188 for the generative judge, and the length-preference shift is -0.023 against -0.217. The confidence-gated cascade, however, escalates between 0.923 and 1.000 of items to the strong judge at the preregistered 0.97 retention bar: calibrated confidence about a near-chance judge has almost nothing to keep. The design: five public preference benchmarks and one hallucination benchmark with real labels, scored under a preregistration frozen before any test item was seen, against generative, reward-model, and trivial baselines, with per-item predictions released.