Calibration method aligns scores without needing ground-truth labels
Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth
Machine LearningArtificial Intelligence
Summary
When combining predictions from different sources, it's hard to compare them if they don't share a common scale or reference. The authors present a way to create a reference scale using synthetic features that represent the concept being scored. This lets them adjust different predictions so they can be meaningfully combined, even without having the true answers or shared labeling systems. Their method works better than simple averaging and performs well on several benchmark datasets.
What this means in practice
- •For machine learning engineers: Combine models producing scores on different scales to create better supervision signals without labeled data.
- •For data integration teams: Align scores from diverse data sources that lack a shared annotation space to enable coherent data fusion.
Authors
Jackson Eshbaugh, Jorge Silveyra
Abstract
We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space. Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion. This reference space is constructed from ordered calibration features that represent the latent concept, providing a common scale on which otherwise incomparable scorer outputs can be aligned. Because our calibration procedure uses the reference space rather than training samples, it is independent of the training set's empirical distribution. Across three benchmark datasets, our framework consistently outperforms uncalibrated averaging and achieves higher primary-metric point estimates on the evaluation metrics than the best individual scorer. Performance relative to sample-dependent baselines varies by domain, with absolute differences below 0.02 on Ames Housing and below 0.01 on Breast Cancer Wisconsin and Wine Quality. After Bonferroni correction, differences remain significant for all three comparisons on Ames Housing and one on Breast Cancer Wisconsin. Additionally, we show that using fewer calibration levels per feature can closely approximate higher-resolution results at substantially lower computational cost. Together, these results support our framework as a viable approach to construct supervision scores when neither ground-truth labels nor a shared annotation space is available.