HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
2026-06-29 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors address the problem of evaluating AI models when expert labels (gold labels) are scarce and expensive, while noisier crowd-sourced labels (silver labels) are abundant but less reliable. They propose a method called HERO that uses past evaluation data to better adjust silver labels, reducing errors and making performance measurements more stable and accurate. HERO leverages historical information to correct biases and lower fluctuations in model evaluations across different rounds. The authors show that their approach works well through simulations and real-world testing on AI model benchmarks.
generative AIgold labelssilver labelscrowdsourcingmodel evaluationbias correctionvariance reductionhistorial dataperformance measurementlabel aggregation
Authors
Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang
Abstract
Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy "silver" labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold labels may have high variance to resolve the model performance gaps that guide practical decisions. Model evaluation has become an ongoing operational practice rather than a one-time exercise, with evaluation rounds repeating across model versions, releases, and content domains. A natural question is whether the previous historical evaluation data can be used to improve each new round of evaluation. We introduce HERO (History Enhanced RObust model evaluation), a novel framework that uses historical data to suppress bias (improve reliability) and reduce variance (improve sensitivity) in model performance evaluation. HERO calibrates silver labelers' performance learned from historical gold annotations, and stabilizes the resulting estimator by anchoring it to covariate information measured with high precision in the historical data. HERO can be broadly applied across multiple common evaluation tasks, and remains valid when only a subset of historical labelers appears in the current round. We establish conditions under which the bias and variance reductions hold, showcase HERO's performance in simulation studies, and demonstrate its effectiveness on real-world model evaluation benchmarking datasets.