Backward conformal prediction improves error estimates under shifted data
Backward Conformal Prediction under Covariate Shift
Machine Learning
Summary
Prediction methods often assume the data they see when learning is similar to the data where they later make predictions. This paper extends a technique called backward conformal prediction to handle situations where this assumption fails because the data distribution changes (covariate shift). The authors improve estimation of how often predictions might be wrong by adjusting for differences using importance weighting and a novel way to estimate these weights based on sorted prediction scores. Their approach leads to more accurate error estimates in various tests, helping predictions stay reliable even when conditions change.
What this means in practice
- •For risk assessment teams: Provide more accurate miscoverage estimates for prediction sets when data distributions differ between training and deployment.
- •For medical data analysts: Improve confidence in prediction intervals when patient characteristics shift between study populations and real-world applications.
Authors
Amin Meghrazi, Yuanyuan Li
Abstract
Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.