Diffusion models improve pretraining for limited data classification
Data-Efficient Classifier Pretraining Through Diffusion Models
Machine Learning
Summary
Training smart computer programs to recognize things often needs lots of examples, but sometimes only a few are available. The authors propose a simple way to teach these programs using diffusion models, a type of technique that helps the program learn from data efficiently. Their method works especially well on medical images and time-related data, often better than other popular methods. This makes it easier to build accurate classifiers when only small amounts of specialized data are available.
What this means in practice
- •For medical imaging teams: Pretrain classification models effectively when limited labeled medical images are available, improving diagnosis tools with less data.
- •For time-series analysts: Improve pretraining of classifiers on time-series data with few labeled examples, enhancing predictions in finance and sensor monitoring.
Authors
Qinyu Li, Yee Whye Teh, Chris Holmes, Joel Jaskari
Abstract
Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.