Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees
2026-08-03 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address problems in tasks where humans have to judge something but there's no clear right answer, and computers alone can't do it well either. They created a two-step method: first, combine many different people's rankings while considering how reliable each person is; second, adjust a computer model’s scores to fit that combined ranking without losing important details. Their math shows this method works better than using just people or just models. Tests on real and simulated data confirm it makes assessments more accurate and reliable.
human judgmentrank aggregationannotator reliabilityisotonic calibrationordinal consistencypredictive modelsrisk boundsconsensus rankingmodel calibrationhuman-centered assessment
Authors
Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang
Abstract
Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth. Existing approaches face a dilemma: methods using only human judgments suffer from heterogeneous expertise and inconsistent rating scales, while methods using only model-generated scores must learn from imperfect proxies or incomplete features. We propose Aggregate-then-Calibrate (AtC), a two-stage framework that combines these complementary sources. Stage-1 aggregates heterogeneous comparative judgments into a consensus ranking using a rank-aggregation model that accounts for annotator reliability. Stage-2 calibrates any predictive model's scores by an isotonic projection onto the order, enforcing ordinal consistency while preserving as much of the model's quantitative information as possible. Theoretically, we show: (1) modeling annotator heterogeneity yields strictly more efficient consensus estimation than homogeneity; (2) isotonic calibration enjoys risk bounds even when the consensus ranking is misspecified; and (3) AtC asymptotically outperforms model-only assessment. Across semi-synthetic and real-world datasets, AtC consistently improves accuracy and robustness over human-only or model-only assessments. Our results bridge judgment aggregation with model-free calibration, providing a principled recipe for human-centered assessment when ground truth is costly, scarce, or unverifiable.