Pre-training on related data overestimates few-shot learning success
Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Few-shot learning means teaching a computer to learn from very few examples. This paper finds that the common way of training computers beforehand on similar but separate data can make few-shot learning seem easier than it really is. The authors show that training on data outside the target domain, even without labels, gives a more realistic view of how well the computer will learn with little data. They also introduce a method to pick the best training data before starting, helping to improve results without bias.
What this means in practice
- •For machine learning engineers: Select training data from appropriate domains before pre-training to improve few-shot learning models with limited target data.
- •For computer vision developers: Implement label-free pre-training methods to reduce data labeling efforts while maintaining few-shot learning performance.
Authors
Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego
Abstract
Few-shot learning is commonly evaluated under protocols that pre-train a model on a large auxiliary set whose classes are disjoint from the target episodes yet drawn from the same visual domain. This paper examines whether such protocols truly reflect low-data learning. We systematically compare no pre-training, class-disjoint in-domain pre-training, supervised out-of-domain pre-training, and label-free out-of-domain pre-training across eight datasets, three few-shot architectures, and multiple way-shot settings. Our results show that class disjointness alone is insufficient to remove the influence of target-domain data. In-domain pre-training improves over no pre-training by 33.41 percentage points on average, whereas supervised out-of-domain pre-training yields 23.75 percentage points, revealing a 9.66-point optimistic bias associated with domain overlap. Although out-of-domain pre-training is more realistic in applications where target-domain data are scarce, its effectiveness depends strongly on the compatibility between source and target domains. We further show that labeled source data are not strictly required, with an augmentation-based label-free strategy reaching an average gain of 27.71 percentage points and closely matching supervised out-of-domain pre-training at 27.97 percentage points. Finally, we introduce a descriptor-based source-selection strategy that estimates source-domain suitability before pre-training, reaching a median gap of only 1.37 percentage points to oracle selection. These findings highlight the need to move beyond in-domain pre-training as the default few-shot evaluation protocol, since it can overestimate performance in realistic scenarios where target-domain data are scarce.