Threshold aware routing cuts costly simulator calls by up to 75 percent
Threshold-Aware Conformal Routing
Machine LearningComputational Engineering, Finance, and Science
Summary
Running detailed simulations can be slow and expensive, so people sometimes use faster approximate models called surrogates. But these surrogates can make mistakes that affect decisions about whether a measurement is above or below an important cutoff. The authors introduce a method called Threshold-Aware Conformal Routing (TACR) that decides when to trust the surrogate and when to run the full simulation, based on how sure the surrogate is about being above or below the cutoff. TACR focuses on making prediction intervals tighter near the threshold, improving decision accuracy and reducing the number of costly simulator runs.
What this means in practice
- •For engineering simulation teams: Reduce expensive simulator runs by dynamically deciding when surrogates suffice for threshold-based decisions.
- •For risk assessment analysts: Improve decision-making reliability by allocating prediction uncertainty near critical thresholds to minimize full computation.
Authors
Shiwei Tan, Huzefa Rangwala, Danielle C. Maddix
Abstract
High-fidelity simulations are essential to scientific and engineering design, but can be expensive to run repeatedly. Learned surrogates offer a faster alternative, yet their higher errors may alter downstream decisions. This accuracy-speed tradeoff creates a need to determine whether a surrogate can be used or the full simulator remains necessary. We study decisions determined by whether a scalar quantity of interest lies above or below a fixed threshold. For each input, we use the surrogate when its conformal interval lies entirely on one side of the threshold and route the input to simulation when the interval intersects it. Standard conformal prediction constructs intervals without reference to the downstream decision threshold: even a narrow interval near the threshold can cross it and trigger simulation, whereas a wider interval farther away can remain entirely on one side and require no simulation. We introduce Threshold-Aware Conformal Routing (TACR), which learns an input-dependent scale using a threshold-aware objective that concentrates interval tightness near the decision boundary. Exact split-conformal calibration on held-out data preserves distribution-free marginal coverage, which also upper-bounds the probability of an incorrect threshold decision that is not routed. Across various scientific and engineering datasets, TACR reduces simulator deferrals by 14-75% relative to standard conformal prediction at the same coverage target. Against a variant without threshold-local weighting but with similar predictor accuracy, TACR further reduces deferrals by 10-24% on four datasets. These results show that optimizing interval allocation for routing can reduce simulator calls without weakening the standard conformal guarantee.