New method improves error control in predictive models for varied inputs

ReCIRC: Rectified Conformal Risk Control

Machine Learning

Summary

Controlling errors in AI predictions can be hard because some cases are easier and some are harder. The authors propose ReCIRC, a way to adjust error limits based on how risky each case is, rather than a single rule for all. This helps avoid being too strict for easy cases or too loose for difficult ones. ReCIRC keeps strong overall error guarantees while better balancing errors across different inputs.

What this means in practice

  • For medical image analysis teams: Control error rates differently for easy and hard cases in medical image segmentation to improve reliability without excessive caution.
  • For machine learning engineers: Tune error boundaries on a per-input basis for multilabel classification models to reduce worst-case group errors while maintaining overall risk levels.

Authors

Bruno Marcondes e Resende, Helton Graziadei, Thiago Rodrigo Ramos, Rafael Izbicki

Abstract

Many applications of black-box predictive models require controlling task-relevant error rates, such as missed lesion pixels in segmentation or missed labels in multilabel classification. Conformal risk control (CRC; Angelopoulos et al., arXiv:2208.02814) gives distribution-free guarantees for such losses, but it calibrates a single threshold shared by all inputs. Because conditional risk varies with the input, this marginal guarantee often overprotects easy cases and underprotects hard ones. We propose ReCIRC (Rectified Conformal Risk Control), which inverts each input's estimated local risk curve to reparameterize the calibrated threshold as a risk budget $a$ representing a common target conditional risk, and then applies CRC unchanged to the resulting family. ReCIRC retains CRC's finite-sample marginal guarantee regardless of the accuracy of the estimated curves, while accurate curves yield approximate conditional risk control and, under additional conditions, asymptotically exact conditional risk control; they also support a risk-calibration diagnostic. Across three synthetic and five real-data settings spanning segmentation, multilabel and multiclass classification, and regression, ReCIRC attained the lowest average worst-group risk and mean positive group excess in every setting, while maintaining marginal risk close to the target, whereas changes in prediction size were application-dependent.