Coreset selection methods often cost more than they save in training
Are Coreset Selection Methods Worth Their Cost?
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
Summary
Coreset selection tries to speed up training by picking a small, representative part of the data. The authors tested many selection methods fairly by counting both the time to choose data and to train models. They found that simple random selection or using all data for fewer training steps often works better than complex selection methods. The cost to pick data is not small and often outweighs the benefit of training on fewer points. Ignoring the selection cost gives a misleading idea of how efficient these methods really are.
What this means in practice
- •For machine learning engineers: Make informed choices about data subset selection by factoring in both selection and training time to optimize training budgets.
- •For data pipeline developers: Avoid implementing costly coreset selection preprocessing that may not reduce overall training time compared to simpler methods.
Authors
Yangze Liu, Zhongyi Han
Abstract
Coreset selection picks a representative subset of the labeled training set to make training cheaper. However, it is usually evaluated by downstream accuracy at a fixed subset size, ignoring both the time spent selecting the subset and the training recipe behind each reported number. We introduce an end-to-end benchmark that standardizes downstream training and charges selection and training to the same auditable wall-clock budget, spanning 4 datasets from CIFAR-10 to ImageNet-1K, 11 selectors, 5 fractions, and 3 seeds, with over 1,500 released runs. Repeated-sampling work has shown that budget-aware evaluation already favors random strategies. Our two budget studies test whether that verdict survives when every selector is granted its most favorable operating point. Across eight wall-clock budget anchors on each of CIFAR-10 and Tiny ImageNet, no anchor is won by a sophisticated selector: every winner is class-balanced random sampling, repeated random sampling, or full-data training. In fixed-budget duels on ImageNet-1K, training on all data for fewer epochs beats every selection strategy we probe while also costing the least. A per-dataset cost audit shows that selection cost is dominated at every scale by a fixed full-dataset scan, so it cannot be amortized away by selecting a smaller fraction, and its absolute size does not extrapolate from one dataset to another. We further quantify when selection does pay back through subset reuse, and document 9 correctness fixes to a widely used codebase, one of which shifts a standard Herding baseline by nearly 6 points. Selection time is not free preprocessing, and an evaluation that ignores it measures the wrong quantity.