Conformal prediction sets minimize length under covariate shift
Finite-Sample Optimal Conformal Prediction Sets under Covariate Shift
Machine Learning
Summary
This paper looks at how to make reliable predictions when the data distribution changes between training and testing, a situation called covariate shift. The authors prove a way to build prediction sets that stay valid while being as small as possible on average, using a combination of known density ratios and special calibration methods. They create practical versions of this ideal method using statistical tools and test them on various datasets, showing improvements in prediction size and speed compared to existing methods. Their approach works well even when the target data has uneven or low-density regions.
What this means in practice
- •For data science teams: Build prediction systems that maintain valid coverage and minimize uncertainty under changing data distributions using weighted conformal methods with known density ratios.
- •For machine learning platform engineers: Incorporate efficient algorithms for conformal prediction sets that adapt better to covariate shifts, improving runtime and predictive confidence in deployed models.
Authors
Zhongyang Li
Abstract
We study conformal prediction under covariate shift with a known covariate density ratio, seeking prediction sets with minimum expected length under the target distribution. For every calibration sample size, we prove that the rule combining the inverse conditional-density score with randomized weighted conformal calibration minimizes this expected length among all rules valid over the full weighted-exchangeability class. We approximate the oracle using quantile and kernel histograms, including forest-weighted estimators, and select estimators and tuning parameters by resampling to estimate calibrated target length. On a bounded response domain, we bound excess length linearly in $L^1$ density error at fixed calibration size, without requiring a uniform positive lower bound on the true density. We also bound selection-pool and resampling errors. Under explicit growth conditions, the selected procedure approaches the same oracle while the calibration size remains fixed. Experiments on 36 known-density configurations and four public regression datasets compare coverage, prediction-set length, and runtime with existing methods. In targeted low-coverage and strong-shift comparisons, inverse-density selection yields shorter mean sets in most cases and lower mean runtime than the evaluated weighted baselines, with some differences in achieved coverage.