TLDChoiceNet: Quantitatively Choosing a Transfer Learning Dataset
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors created a model called TLDChoiceNet to help decide which dataset is best to use for transfer learning when you have a smaller fine-tuning dataset. Their model predicts how well a model will perform after fine-tuning on different datasets. They also developed two new ways, called distribution distance and average class correlation, to pick the best transfer dataset without needing extra labels. Their work shows that simple statistics about datasets and pretrained features can explain how well transfer learning works.
transfer learningfine-tuningImageNetdataset selectiontest accuracy predictiondistribution distanceaverage class correlationResNet50mean squared error
Authors
Jing Ning, James D. Braza
Abstract
Transfer learning is particularly useful in settings with limited training data, and within image classification it is common to transfer learn upon massive datasets like ImageNet , CIFAR-100, or COCO . Qualitatively, it seems a transfer learning dataset should have both more classes and more examples per class than the fine tuning dataset; however, a quantitative method to choose the best transfer learning dataset does not currently exist. In this paper, we design TLDChoiceNet, a model to choose the best transfer learning dataset given a fine tuning dataset by predicting the test-set accuracy after fine-tuning. A simple version 1 achieves 0.154 MSE on the test dataset, while a version 2 leveraging an ImageNet pre-trained ResNet50 v2 embedding with per-class information attains a 5X lower MSE of 0.031. We further design two metrics that enable an unsupervised method of choosing an optimal transfer learning dataset: distribution distance (DD), which linearly regresses against fine-tune accuracy with an R2 of 0.89, and average class correlation (ACC), which improves the R2 to 0.97. Our results underscore that a dataset's low-level statistics can explain the transfer learning effect, and that using a pre-trained ImageNet can embed different classes further apart in latent feature space.