Model acceptance gate cuts false updates in clinical settings
BRIDGE: An Acceptance Gate to Block False Improvements in Model Updates, with a Retrospective Clinical Evaluation
Machine Learning
Summary
Models that make decisions, like in healthcare, get updated often. Sometimes updates look better when tested on old data but actually perform worse on new cases, causing mistakes. The authors present BRIDGE, a method that carefully tests model updates using separate parts of the data to avoid wrong approvals. This method greatly reduces false improvements while increasing the true benefit of updates. It was tested on many clinical and synthetic datasets with strong results.
What this means in practice
- •For clinical data teams: Screen model updates in clinical decision systems to avoid adopting harmful or useless changes.
- •For machine learning engineers: Improve deployment decisions by systematically reducing acceptance of false positive model improvements during development.
Authors
Dwi Sunaryono
Abstract
Models are updated repeatedly, and an update that looks better on validation data but not on new data is a false improvement that can misguide clinical decisions. The naive rule accepts every positive validation gain and is optimistic when the same data choose and judge the update. BRIDGE is a model-agnostic acceptance gate that decides each update from predictions and the available labels, with fixed settings. Each fold of the data judges the update chosen on the other folds (cross-fitting) to reduce selection bias; acceptance requires the gain's lower bootstrap bound to exceed a margin. The four settings are (A) 11 new datasets, (B) synthetic tasks, (C) 12 retrospective clinical tasks, and (D) 39 supporting datasets. The false-improvement rate falls from 19.1% (naive) to 1.1% (BRIDGE) in (A), 20.7% to 1.7% (B), 20.0% to 0% (C), and 12.5% to 0.5% (D); every 95% confidence interval of the reduction over datasets or tasks excludes zero (not over the clinical sources). An equal-size random draw from the naive acceptances contains more false improvements in (A)-(E) (post hoc, p<0.001). On the OpenML collection (E) of 14 new datasets, the false-improvement rate falls from 25.6% to 1.0%. BRIDGE removes 97% to 100% of the false improvements that the naive rule accepts, and its mean gain per deployed update is 1.8 to 3.1 times the naive rule's. The 95% upper bound on the false-improvement rate is below 5.5%: formal in (B) for independent tasks (3.56%), and conditional on the real data tested (0.33% to 5.40%).