Correcting uncertainty estimates under changing label distributions
Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
Machine Learning
Summary
Predicting with confidence intervals can fail when the kinds of outcomes change between training and use, a problem called label shift. The authors propose a new way to adjust uncertainty estimates that accounts for this shift by modifying Bayesian predictive scores. This adjustment helps produce tighter confidence intervals without losing reliability, even when the distribution of labels differs. Their method works generally for models with accessible predictive densities and they tested it on molecular property prediction tasks.
What this means in practice
- •For pharmaceutical data scientists: Generate reliable prediction intervals adjusted for changing label distributions in molecular property models.
- •For financial risk analysts: Improve uncertainty estimates under shifts in market outcome frequencies using label-shift-adjusted Bayesian scores.
Authors
Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba, Seungjin Choi, Hyunjin Shin
Abstract
Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the \emph{Label-Shift-Adjusted Bayesian Score} (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.