Papers for

machine learning evaluation teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Combining human and AI feedback to better rank language models

Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback

Abstract: Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular $\sqrt{T}$-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.

Mon 9 NovMachine Learning
The gist
Ranking language models by using human feedback helps decide which model works best, but getting lots of human evaluations is expensive. The authors propose a method called PP-Rank that mixes human feedback with predictions from AI judges to improve ranking accuracy without needing many human reviews. Their approach handles tricky dependencies from collecting data step-by-step and adjusts how to collect feedback more efficiently. Experiments show this method can save evaluation costs while keeping results trustworthy.
Open → 2611.13186v1

CloseSort method finds exact maximum likelihood rankings from pairwise data

Close and Sort: Exact Maximum-Likelihood Aggregation of Cyclic Rank-Two Preferences

Abstract: Pairwise probabilities arise in rank aggregation, recommendation, and human or LLM evaluation, yet converting them into a maximum-likelihood list is a weighted Kemeny problem and is NP-hard in general. Known exact algorithms for rank-two tournaments use only comparison signs and do not optimize this likelihood. We show that the weighted problem is exactly solvable when the skew-symmetric log-odds matrix $W_{ij}=\log(p_{ij}/p_{ji})$ has rank two, including when the induced preferences contain cycles. Writing $W_{ij}=\det(h_i,h_j)$ and adding the closing vector $h_0=-\sum_i h_i$ turns the objective into twice the signed area of a closed polygon. This geometry yields CloseSort, which polar-sorts the $n+1$ edge vectors, cuts after $h_0$, and returns a global maximum-likelihood order in $O(n\log n)$ time from the factors. The same geometry identifies the runner-up and gives a stability certificate beyond exact rank two. When the smallest determinant $δ$ of two consecutive edges in the closed angular cycle is positive, it is exactly the best-to-runner-up log-likelihood gap. For $W=W_0+E$, the CloseSort order of $W_0$ has regret at most $\sum_{i<j}|E_{ij}|$ on $W$ and remains optimal whenever this sum is below $δ$, a worst-case tight threshold. In 5,120 determinant-logit simulations, increasing the observations per pair from 4 to 64 raises the fraction on which rank-two projection outperforms the empirical plug-in optimum from 68.6% to 85.7%. On 3,600 Arena-Hard LLM-judge profiles drawn from 63 systems and ten judges, the leading rank-two component captures 89.2% of the log-odds energy. On profiles containing a directed majority triangle, CloseSort reduces normalized regret by 44% relative to Borda and improves the mean normalized objective over each of eight baselines. With factors supplied, it is 8,407 times faster than exact dynamic programming on 18-item rankings.

Sat 7 NovData Structures and AlgorithmsInformation RetrievalMachine Learning
The gist
Finding the best overall ranking from pair-by-pair preferences is usually very hard and slow. The authors show that if the underlying data matrix has a special two-dimensional structure, this ranking problem can be solved exactly and quickly. They created CloseSort, an algorithm that uses a geometric approach to order items in roughly n log n time. CloseSort also measures how stable the ranking is and works well even when there are cycles in preferences, improving on older methods in speed and accuracy.
Open → 2611.10538v1

Agora method improves aggregating opinions of large language model judges

AGORA: Reliability-Aware Aggregation of LLM Judges via Item-Response Theory

Abstract: LLM-as-a-judge is a common substitute for costly human evaluation, and aggregating several judges is the standard remedy for any single judge's bias. Judge pools mix judges of very different reliability, while the human-labeled sets that validate them hold a few hundred items or fewer. In this regime the per-item mean weights weak judges as heavily as strong ones, and confusion-table models such as Dawid-Skene must estimate a per-judge error table from too few items. We propose AGORA (Aggregation Of Reliability-weighted Assessments), an item-response-theory aggregator that places every item on one latent quality scale and describes each judge by a sensitivity parameter and ordered rating thresholds, so judge reliability is learned from all items jointly and without human labels. Across three benchmarks, AGORA matches or exceeds Dawid-Skene on large evaluation sets, and its advantage widens as the set shrinks toward a few hundred items. It remains accurate as pools grow more uneven, a label-free dispersion statistic indicates in advance when trust weighting helps, and its learned trust weights track per-judge human validity.

Tue 3 NovArtificial IntelligenceComputation and Language
The gist
When many AI models called large language models (LLMs) act as judges for tasks usually done by humans, their opinions can vary in accuracy. The authors propose a way to weight each AI judge’s opinion based on reliability, learned without needing examples judged by humans. Their method, AGORA, puts every task on a quality scale and figures out how sensitive and trustworthy each judge is by looking at all tasks together. Tests show AGORA works better especially when the number of tasks is small or judges differ a lot in skill.
Open → 2611.05556v1

Automatic scores help reduce human evaluation effort but have limits

Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains

Abstract: Automatic judges offer inexpensive information about model outputs, but an evaluation team must decide how to turn that information into fewer human annotations. Scores can guide sampling, correct an estimate, or support adaptive acquisition, and these uses may capture overlapping gains. We compare their incremental value on fixed evaluation pools from three preference datasets, and 19 attributed-question-answering systems. Matched comparisons use the same scores, account for pilot labels and budget rules, and separate acquisition from estimation. Score stratification is a strong reference: after ten-bin stratification, the best fixed local linear correction offers only 1.00--5.27% further MSE reduction in the primary configurations. Unequal allocation offers larger gains in some pools, but can increase error when confident scores conceal variable human outcomes. Comparison on AttributedQA, shrinking learned quotas toward score-only quotas lowers pooled MSE for 17 of 19 systems relative to the score-only procedure, but ordinary pooled 95% intervals cover only 88.09--93.54%. The same shrinkage improves on the score-only reference in only one of eight preference configurations. Ranking experiments further show that lower individual MSE need not improve system selection. These findings support a simple starting procedure, proportional sampling within score strata followed by a stratified human mean, and identify what additional evidence is needed to justify correction or adaptation for the intended evaluation output.

Wed 28 OctComputation and Language
The gist
Human evaluation of AI outputs is expensive, so automatic scores can guide which outputs should get human review or adjust estimates. The authors compare ways to use these automatic scores, like sorting samples or correcting errors, to see which save the most effort while keeping accuracy. They find a simple method—sampling proportionally within score groups and averaging human ratings—is often nearly as good as more complex methods. More advanced corrections help only slightly, and sometimes scores can hide real human rating differences, causing errors. These insights help decide when and how to include automatic judging in human evaluations.
Open → 2610.38282v1

Ranking models reliably during running benchmarks without errors

Rank Confidence Sequences:Anytime-valid Leaderboards

Abstract: Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $α$, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.

Sat 26 SeptArtificial Intelligence
The gist
Leaderboards show how well different models perform on a test, but checking ranks repeatedly while tests are still running can lead to mistakes. The authors created a way to safely update rankings at any point without increasing errors, using mathematical methods that consider all possible score dependencies. This means you can stop testing a model early when confident about its rank or check the leaderboard anytime without losing accuracy guarantees. Their method works even when models are tested on the same data and keeps error rates at a chosen level.
Open → 2609.32211v1