Model internal features improve selection of images to label for training
From internal representations to model improvement through prediction errors
Computer Vision and Pattern RecognitionMachine Learning
Summary
Choosing which images to label can strongly affect how well a machine learning model improves, especially when labels are expensive. The authors show that looking at the model's own internal features—what it has learned so far—can help pick the best images to label next. They link these internal features to where the model makes errors and how fixing those errors will improve performance. Their method outperforms simpler approaches that only consider how rare features are, leading to better results after retraining.
What this means in practice
- •For machine learning engineers: Improve active learning pipelines by selecting images for labeling based on the model’s internal features tied to impactful prediction errors, enhancing retraining efficiency.
- •For data annotation teams: Prioritize image labeling tasks by identifying which unlabeled images will lead to the greatest model performance improvement, reducing labeling costs.
Authors
Yushi Nakaya, Kenichi Higuchi, Shuichi Ishida
Abstract
With limited annotation budgets, choosing which images to label determines how much a model improves. Data-selection methods that use features from a separately trained model, or scene descriptions written by vision-language models, have been successful, but those signals do not directly capture changes in the model being improved. The target model's own internal features reflect what it has learned so far and change with retraining, making them a natural cue for choosing the next training data. However, feature rarity alone does not reveal the errors that matter for performance. Here we link internal features to prediction errors and their expected impact on performance and select images for labeling and retraining without using labels for candidate images. We evaluated the method with an object detector on two datasets and two pairs of random seeds. Adding internal features improved the identification of prediction errors in 15 of 16 conditions. When performance was averaged over successive labeling rounds, the method outperformed selection based only on feature rarity in all four evaluation settings and ranked among the top two of six methods. With other conditions held fixed, performance after retraining was again higher than with rarity-based selection, even though the latter collected more errors. With longer retraining, the proposed method ranked first among six methods. These results suggest that linking a model's internal features to its errors and their effects on performance may help select training images that improve performance, thereby allowing the model's current state to guide which images are labeled next.