Computer model calibration controls prediction errors reliably
Prediction Uncertainty Controlled Computer Model Calibration
Machine Learning
Summary
Computer models are often adjusted to better predict real-world events, but traditional methods don't always guarantee how accurate these predictions are. The authors present a new two-step approach that better controls the chances of prediction errors by learning from simulated data and carefully selecting prediction ranges. They tested their method on synthetic data and a disease outbreak model, showing it produces more precise and reliable prediction intervals than previous techniques. This also helps identify when the underlying models may not fully capture the real situation.
What this means in practice
- •For epidemiological modelers: Improve the reliability of epidemic forecasts by better quantifying and controlling predictive uncertainty in mechanistic disease models.
- •For complex system analysts: Calibrate simulation models with controlled prediction errors even when data or model assumptions are imperfect, helping to identify model limitations.
Authors
Ryan Heejoon Byun, Dayeon Yoon, Won Chang
Abstract
Computer-model calibration is usually undertaken in service of prediction, yet neither Bayesian Kennedy-O'Hagan (KOH) calibration nor amortized simulation-based inference is built to control the predictive error of the calibrated simulator. When parameters are weakly identified or the data-model discrepancy is misspecified, the simulator-output bands induced by a nominal posterior level can be far from nominal. We propose a two-layer framework that targets predictive miscoverage directly. First, a conditional flow-matching model learns an amortized posterior for the calibration parameters, trained on simulator outputs perturbed with Gaussian-process discrepancies of varying structure. Second, conformal risk control arranges posterior draws for the target observation into nested candidate sets and selects the smallest one whose fixed-rank pointwise bands attain a prescribed time-averaged miscoverage on a synthetic calibration sample. Under exchangeability of calibration and target observations, the bands control expected miscoverage in finite samples. In a synthetic benchmark, the proposed bands are roughly 20% the width of KOH-MCMC bands with miscoverage near the 10% target. In calibrating an age-structured SEIR model to the 2010 Malawi measles outbreak, they are again sharper and near-nominal, and the weeks they fail to cover coincide with the asymmetric epidemic decline, exposing structural inadequacy of the mechanistic model.