Accounting for uncertainty improves defect detection in medical device manufacturing
Epistemic Uncertainty-Aware Defect Detection for Quality Control in Medical Device Manufacturing
Artificial Intelligence
Summary
Detecting manufacturing defects in medical devices is crucial but challenging, especially when the decision system is unsure. The authors studied a method that lets the AI system 'say I don’t know' by recognizing when it is uncertain about its predictions. By refusing to decide on uncertain cases and leaving them for further human inspection, the system can reduce mistakes and improve overall accuracy. Their tests on both fake and real device data showed much better reliability when uncertainty was considered.
What this means in practice
- •For medical device manufacturers: Control automated defect detection reliability by rejecting uncertain predictions to improve safety and reduce errors in device quality inspections.
- •For quality assurance teams: Identify device reports requiring extra human review by using uncertainty-aware machine learning models to prioritize inspection resources.
Authors
Raham A. Butt, Marco Romanelli, Roche C. de Guzman
Abstract
Objective: We investigate whether accounting for epistemic uncertainty can improve the reliability of automated defect detection in medical device manufacturing. Methods: We consider a machine learning framework that operates on heterogeneous manufacturing and device-report data represented with Knowledge Graphs. To mitigate errors arising from uncertainty in the decision model, we analyze a principled rejection strategy to abstain from predictions whose estimated epistemic uncertainty exceeds a specified threshold. We evaluate the approach using standard synthetic benchmarks and real-world medical device report data. Results: The theoretical results establish the validity of the method characterizing the regimes under which it is expected to be effective. Empirically, the rejection strategy enables explicit control of coverage, that is, the proportion of samples for which the model issues predictions, while improving performance on the retained samples. On 266,170 real-world FDA MAUDE device reports, a 10% abstention rate reduces classification error by 48%, and more aggressive rejection (approximately 70% coverage) yields near-perfect accuracy on the retained samples. On standard synthetic manufacturing benchmarks, abstaining on 9% of the decisions, our approach reduces the risk up to 63% compared with the standard no-abstention approach. Conclusions: Abstaining from predictions with high epistemic uncertainty can provide a practical tool for controlling the reliability of machine learning-based defect detection, especially in high-stakes medical device manufacturing applications. Significance: Uncertainty-aware defect detection may support safer and more reliable quality assurance in medical device manufacturing by identifying cases that require additional inspection rather than issuing potentially harmful predictions.