Adaptive rules improve stopping decisions in uncertain classifications
How Many Posterior Samples? Calibrated Stopping for Adaptive Sensing
Computer Vision and Pattern Recognition
Summary
When computers try to decide what something is by looking at data piece by piece, they guess and update their confidence as they go. The authors explore better ways to decide when the computer should stop looking for more data and make a final choice. They found that simple rules sometimes stop too soon or too late, so they introduced ways to carefully balance how often mistakes happen. By testing on handwriting images, they show their approach can save lots of effort while keeping decisions accurate.
What this means in practice
- •For machine learning engineers: Optimize the number of uncertainty samples needed before making classification decisions to reduce computational costs without losing accuracy.
- •For medical imaging teams: Improve adaptive scanning processes by stopping data collection at the right time, reducing patient exposure and scan time.
Tested on one dataset.
Authors
Vincent Corlay, Andriy Enttsel
Abstract
In classification-oriented adaptive sensing, posterior samples characterize uncertainty at the current measurement state and can serve two roles: they may guide the next sensing direction, while their class labels provide votes for the candidate classes and determine whether sensing should continue. We focus on the stopping layer that turns these votes into a declaration, without modifying the posterior sampler or sensing directions. A natural plug-in rule declares when the observed vote share exceeds a threshold. We show that this threshold is not itself a confidence guarantee: when the underlying vote mass equals the threshold, the plug-in rule declares about half the time. As alternatives, we calibrate a fixed-sample rule and a finite-horizon sequential rule to a prescribed false-declaration probability, and study exact curtailment, which stops a fixed-pool rule once its final verdict is forced. We then derive how one-round declaration probabilities determine posterior-sample cost and classification accuracy along a sensing path. On MNIST with DDRM and a fixed PCA-guided probe sequence, curtailment saves up to 62% of posterior samples. Among the evaluated rules at matched operating points, sequential stopping reduces the cost the most. At a high accuracy, that same sequential rule can trade more posterior samples for fewer measurements.