Engineering models face challenges quantifying uncertainty under changing conditions
Benchmarking uncertainty quantification for engineering regression under graded distribution shift
Machine Learning
Summary
Machine learning models used in engineering often make predictions outside the situations they were trained for, where knowing how uncertain those predictions are is very important. The authors tested 12 uncertainty measurement methods on several engineering datasets, including some that changed in tricky ways. They found that most methods work well when conditions are similar to training but struggle when conditions shift significantly. They suggest better testing plans and recommendations for using uncertainty methods in engineering settings.
What this means in practice
- •For industrial machine learning teams: Evaluate uncertainty methods under real deployment shifts using the paper’s graded shift framework and recommended data splits for better reliability.
- •For sensor deployment engineers: Use the findings to choose uncertainty quantification methods that maintain coverage and handle abstention in sensor systems experiencing environmental changes.
Authors
Ahmad Lawal, Rahim Pasbanigoloojeh, Asmau Lawal Bilbis
Abstract
Machine learning models in engineering are often used beyond their training conditions, where reliable uncertainty quantification (UQ) matters most. Existing comparisons of regression UQ methods rely on random splits, which assess calibration only in distribution. We introduce a three-rung shift ladder (in distribution, soft shift and hard shift) built along physically meaningful shift variables, and evaluate 12 methods from five families on seven public engineering datasets: five extrapolation settings, an unseen turbofan fault mode and a winter air-quality sensor deployment. All tuning uses source data only. In distribution, conformal methods cover at least 0.946 of test points on every dataset. Under hard shift, mean coverage falls to 0.40-0.75 across the 11 methods that always return an interval, and no systematic difference in method ranks is detected (Friedman p = 0.63). Weighted conformalised quantile regression stays near nominal on the points it answers, but abstains on 85\% of hard-shift points on average and covers only 0.67 of the few Concrete points it answers. Source-validation gains predict improvement in distribution and under soft shift, but not under hard shift (Spearman's rho = 0.04). Engineering evaluations should split calibration data by deployment unit, report subgroup coverage and sharpness, and treat abstention as an explicit operational outcome.