Active learning creates sparse winning tickets with minimal extra cost
One Loop, Two Gains: Can Active Learning win the Lottery for Free?
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Training computer vision models usually takes a lot of time and computing power. The authors found a way to combine two techniques—active learning and iterative pruning—so that they can create smaller, simpler models called winning tickets without extra retraining costs. These smaller models perform just as well as big ones and can speed up training and decision-making steps in machine learning. Their approach works well across different model types and image tasks, making it easier to build efficient, accurate AI systems.
What this means in practice
- •For machine learning engineers: Produce smaller models during each active learning cycle that retain accuracy, reducing computation time for retraining and data selection.
- •For edge device developers: Deploy highly sparse deep learning models created during active learning to run accurate image classification with lower resource use.
Authors
Benedikt Tscheschner, Eduardo Veas, Marc Masana
Abstract
The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initialization, match the accuracy of the full dense network. The predominant method for discovering such tickets, iterative magnitude pruning, alternates pruning with full retraining from scratch until convergence over many cycles. Similarly, deep active learning also retrains a model from scratch after each acquisition round as new labels become available. Despite this shared reliance on iterative retraining with a substantial computational overhead, the two paradigms have been studied separately. We observe that the iterative training loop inherent to pool-based active learning already provides the exact computational structure that iterative magnitude pruning exploits, and propose Improve & Prune (I&P), a method that integrates magnitude pruning into each active learning retraining cycle at practically no additional cost. This raises a key empirical question: can iterative magnitude pruning produce winning tickets under the non-stationary data regime of active learning? We investigate this question across multiple acquisition functions, architecture families, and image classification datasets, including an active fine-tuning scenario. Our results demonstrate that I&P yields sparse, deployable models at each active learning iteration. Those match the accuracy of their dense counterparts at sparsities up to 95%, effectively obtaining winning tickets as a byproduct of the active learning pipeline. These per-iteration sparse models can address two computational bottlenecks - per-round model retraining and acquisition scoring over the unlabeled pool - that currently prevent the practical adoption of DAL on large architectures and large unlabeled pools.