Synthetic labels improve conditional coverage in conformal prediction

How Synthetic Labels Improve Conformal Prediction: A Perspective on Conditional Coverage

Machine Learning

Summary

Conformal prediction is a way to make reliable predictions with guaranteed accuracy on average, but it can struggle to estimate uncertainty well for specific kinds of inputs when only a small set of trusted data is available. The authors investigate using synthetic labels generated cheaply from domain or language models to help improve these predictions. They combine synthetic and trusted data in a new method called prediction-powered quantile learning, which better estimates where uncertainty lies and produces tighter prediction intervals. Their experiments show that adding synthetic labels leads to better uncertainty estimates without losing overall reliability.

What this means in practice

  • For machine learning engineers: Improve uncertainty estimates in regression models when only limited trusted data is available by incorporating synthetic labels from domain or language models.
  • For data labeling teams: Enhance calibration of predictive models by augmenting scarce labeled data with cheaper synthetic labels for better conditional coverage and tighter prediction sets.

Authors

Qianyi Chen, Bo Li

Abstract

Conformal prediction provides distribution-free finite-sample marginal coverage, but post-hoc calibration data may be too scarce to learn how uncertainty varies across inputs. Meanwhile, abundant covariates can often be labeled cheaply by domain models or general-purpose language models. We study whether these synthetic labels can improve conditional coverage when only a small trusted sample is available. Building on score-quantile regression, we introduce prediction-powered quantile learning: a synthetic-labeled pool estimates pinball risk, paired trusted and synthetic outcomes correct its bias, and an independent trusted split performs final conformalization. Profiling pinball risk over scalar corrections reveals that population conditional-coverage error is its functional gradient; the corresponding Hessian removes global shifts and weights remaining shape error by boundary density. Composing this geometry with prediction-powered learning yields a three-resource expansion and a benefit--cost rule for synthetic power. Across eight regression benchmarks, synthetic-powered quantile learning substantially improves downstream conditional coverage while preserving marginal validity and producing more compact prediction sets. A human-rating study finds similar gains from external LLM labels and exposes a quality--quantity--cost tradeoff.
View PDFOpen arXiv