Combining human and AI feedback to better rank language models
Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback
Machine Learning
Summary
Ranking language models by using human feedback helps decide which model works best, but getting lots of human evaluations is expensive. The authors propose a method called PP-Rank that mixes human feedback with predictions from AI judges to improve ranking accuracy without needing many human reviews. Their approach handles tricky dependencies from collecting data step-by-step and adjusts how to collect feedback more efficiently. Experiments show this method can save evaluation costs while keeping results trustworthy.
What this means in practice
- •For machine learning evaluation teams: Improve ranking precision of language models using limited human feedback combined with AI judge predictions, reducing evaluation costs.
- •For automated content moderation teams: Better prioritize model comparisons to identify top-performing language models while minimizing costly human review effort.
Authors
Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
Abstract
Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular $\sqrt{T}$-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.