Optimal schedules improve labeling decisions for machine learning tasks

Towards optimal selection schedules for online active learning

Machine Learning

Summary

Labeling data can be expensive and time-consuming, especially when machines learn from a continuous stream of information. The authors studied how to best decide which data points to label to improve learning accuracy while limiting costs. They found that choosing samples with certain confidence levels—called margins—in a particular way can optimize learning outcomes. They tested their method on synthetic and medical image data and also explored how it applies to neural networks, showing that simple approaches might not always work best.

What this means in practice

  • For machine learning engineers: Optimize data labeling in continuous learning systems by using margin-based sample selection schedules that reduce labeling costs and improve model accuracy.
  • For medical image analysts: Enhance training of diagnostic AI by selectively labeling only the most informative medical images as guided by optimized selection threshold schedules.

Authors

Hugo Cui, Francesca Mignacco, Francesco Mori

Abstract

We consider the problem of online linear classification, where a stream of unlabeled samples is available, and annotation is costly. We investigate the optimal query policy, that prescribes whether and when a sample should be labeled and used for a training step, or instead discarded, so as to minimize a given objective. We focus on the realizable case, under Gaussian design, in the high-dimensional regime where data dimension and number of unlabeled samples grow proportionally. We consider two objectives: 1) the generalization error under a labeling budget, and 2) a cumulative cost function where mistakes have unit cost and queries cost $Δ$. We first show that the myopic policies that greedily minimize at all times the increment in the objective correspond to either small or intermediate-margin policies, that prescribe the selection of samples whose margins fall within certain intervals that shrink with time. Within this class, we numerically ascertain the optimal schedules for the selection thresholds that minimize the final values of the objectives. Taking inspiration from the myopic policy for the cumulative cost, we design a very simple query policy schedule that we compare to other proposed heuristics, on synthetic data and real medical image datasets. Finally, we discuss myopic policies for two-layer neural networks, and exemplify how naive margin-based policies can cease to be optimal.