Sharp and stable verification of predictors without labeled data
Sharp and Stable Label-Free Calibration Certificates from Logical Coherence
Machine Learning
Summary
Sometimes, you want to check if a system's predictions are trustworthy, but you don’t have the right answers to compare against. The authors developed a new way to measure how well predictions are calibrated using only the predictions themselves, no labels needed. Their method can provide a minimum level of how much the predictions deviate from being perfectly calibrated, and these measurements remain reliable even when predictions change. They also tested their approach on real data and found that it consistently flagged imperfect calibration.
What this means in practice
- •For machine learning engineers: Validate the reliability of AI models' probabilistic outputs without needing labeled test data.
- •For data-driven decision teams: Get trustworthy flags about how well real-time predictions match expected outcomes even when true labels are missing.
Authors
Takahiro Kato
Abstract
Query-conditioned probabilistic predictors answer logically related questions one at a time, and labels to check their calibration are often missing. For each restriction on what a bettor's stakes may use, a minimax theorem identifies, for finitely supported laws of the answers, the sharp label-free lower bound on the corresponding calibration residual. On bundles of different events its mean, marginal and joint stages can separate strictly, and a bundle's bound cannot be attributed to one question without assumptions on the others. The sharp marginal bound is discontinuous in the answers, and no label-free lower bound continuous in the Wasserstein distance stays within a constant factor of it. Lipschitz stakes give a stable bound, sharp for a smooth calibration residual and, for every positive Lipschitz constant, positive exactly when the sharp marginal bound is. On i.i.d. streams, projected online gradient ascent at fixed nodes gives fixed-horizon certificates whose shortfall is bounded by grid, learning and statistical terms, and predictable betting gives anytime-valid ones. In a preregistered study of four predictors on two data sets, all seven confirmatory certificates were positive (the eighth pair did not pass the pilot). A certificate shows only that calibration is at least this bad; zero means "not refuted".