Automatic scores help reduce human evaluation effort but have limits
Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains
Computation and Language
Summary
Human evaluation of AI outputs is expensive, so automatic scores can guide which outputs should get human review or adjust estimates. The authors compare ways to use these automatic scores, like sorting samples or correcting errors, to see which save the most effort while keeping accuracy. They find a simple method—sampling proportionally within score groups and averaging human ratings—is often nearly as good as more complex methods. More advanced corrections help only slightly, and sometimes scores can hide real human rating differences, causing errors. These insights help decide when and how to include automatic judging in human evaluations.
What this means in practice
- •For machine learning evaluation teams: Reduce human annotation costs by using proportional sampling within score strata followed by stratified averaging of human ratings to estimate system quality efficiently.
- •For chatbot developers: Improve the cost-effectiveness of human preference evaluations by using automatic score stratification to guide sampling and estimation.
Authors
Yang Liu, Ran Zou, Zhengtao Gui, Chao Xue
Abstract
Automatic judges offer inexpensive information about model outputs, but an evaluation team must decide how to turn that information into fewer human annotations. Scores can guide sampling, correct an estimate, or support adaptive acquisition, and these uses may capture overlapping gains. We compare their incremental value on fixed evaluation pools from three preference datasets, and 19 attributed-question-answering systems. Matched comparisons use the same scores, account for pilot labels and budget rules, and separate acquisition from estimation. Score stratification is a strong reference: after ten-bin stratification, the best fixed local linear correction offers only 1.00--5.27% further MSE reduction in the primary configurations. Unequal allocation offers larger gains in some pools, but can increase error when confident scores conceal variable human outcomes. Comparison on AttributedQA, shrinking learned quotas toward score-only quotas lowers pooled MSE for 17 of 19 systems relative to the score-only procedure, but ordinary pooled 95% intervals cover only 88.09--93.54%. The same shrinkage improves on the score-only reference in only one of eight preference configurations. Ranking experiments further show that lower individual MSE need not improve system selection. These findings support a simple starting procedure, proportional sampling within score strata followed by a stratified human mean, and identify what additional evidence is needed to justify correction or adaptation for the intended evaluation output.