Rubric calibration improves accuracy of large language model document rankings
Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory
Information Retrieval
Summary
Deciding which documents show up first in search results is tricky because human labels that judge relevance are limited and inconsistent. The authors created a method called Rubric-Calibrated Preferences that combines detailed comparisons within searches and clear yes/no standards across searches to better rate document relevance. This method uses a testing theory to put all these ratings on the same scale, improving agreement with human judgments and distinguishing closely ranked results more effectively. Their new metric, RCP-nDCG, better separates good documents from less useful ones compared to traditional methods.
What this means in practice
- •For search system developers: Evaluate and improve document ranking quality by using rubric-calibrated LLM preferences for more consistent ranking metrics across queries.
- •For content moderation teams: Prioritize content more reliably by employing calibrated relevance scores that better agree with human assessment of usefulness.
Authors
Fabian David Schmidt, Donato Crisostomi, Carlos Lassance, Nils Reimers
Abstract
Rerankers decide which documents users and LLMs see, yet their standard metric, nDCG, relies on human relevance labels that are costly, sparse, noisy, and discretely graded. As rerankers approach each other in quality, nDCG on these labels therefore increasingly fails to separate them. LLM judges could supply dense labels. Relative judgments within one query tell even close candidates apart, yet their scores share no scale across queries. Absolute grades share one scale but are too coarse to distinguish documents of similar relevance. We propose Rubric-Calibrated Preferences (RCP), which combine both kinds of judgment. A listwise Bradley-Terry tournament orders each query's documents, and a rubric of yes/no criteria of increasing stringency provides an absolute standard. Item Response Theory (IRT), which scores test-takers based on their answers to common questions, then uses the shared criteria to put all queries' tournament scores on one scale. RCP's retrieval metric, RCP-nDCG, replaces nDCG's discrete labels with the resulting calibrated relevance probabilities. Against blind grades from 46 external annotators, calibration raises the correlation between a query's mean score and its mean human grade from 0.538 to 0.795. The probabilities rank a useful document above a non-useful one with probability 0.910 (AUC, chance 0.5), versus 0.651 for the benchmark labels. When the annotators' grades prefer one of two rerankers and exactly one metric agrees, that metric is RCP-nDCG in 72.4% of 185 comparisons (chance about 53%). On TREC-DL, RCP-nDCG sides with NIST assessors' grades on every reranker pair that these grades separate significantly. RCP-nDCG also resolves many of nDCG's ties and separates 1.9 times as many reranker pairs on NanoBEIR. Rubric calibration thus turns relative LLM judgments into dense relevance labels that are comparable across queries and agree with human judgment.