Pretraining and distillation improve single-cell classification more than architecture
Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification
Computer Vision and Pattern Recognition
Summary
Classifying single cells without labels using deep learning is tricky because different model types seem to perform differently. The authors studied popular deep learning models on a tough microscope image dataset and found that how models are trained before fine-tuning and knowledge distillation matter more than the model type itself. Smaller pretrained models can even beat larger models trained from scratch. This means the way models learn beforehand and share knowledge is more important than picking a specific architecture.
What this means in practice
- •For biomedical imaging teams: Use pretrained and distilled models to improve label-free single-cell classification efficiency and accuracy in microscopy workflows.
- •For machine learning engineers: Develop compact and high-performing single-cell classifiers by focusing on pretraining and knowledge distillation rather than architecture changes.
Authors
Philip Graemer, Giuseppe Di Caprio
Abstract
Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicting conclusions about CNNs versus transformers. We present a controlled benchmark on LIVECell phase-contrast microscopy data using source-image-disjoint train/validation/test splits to prevent parent-image leakage and matched optimisation, augmentation, and evaluation protocols across EfficientNet, Vision Transformer (ViT), and EVA-02 models. This allows the effects of architecture, pretraining, fine-tuning, tokenisation, and distillation to be disentangled. We find that the previously reported CNN advantage is largely explained by pretraining rather than architecture: the smallest pretrained model outperforms the strongest model trained from scratch despite far fewer parameters. Pretraining improves macro-F1 by 3-4 points, while the gap between the best pretrained CNN and transformer is below 0.5 points. Architectural choices nevertheless matter: ViT-S/8 outperforms ViT-S/16 and matches the four-times-larger ViT-B/16 at a quarter of the parameters, showing that finer tokenisation benefits small cell crops. Conversely, layer-wise learning-rate decay, central to the EVA-02 fine-tuning recipe, degrades performance, highlighting that transfer heuristics from natural-image recognition may not generalise to microscopy. Finally, knowledge distillation substantially improves the deployment frontier: compact EfficientNet-B0 students distilled from teacher councils outperform every individually trained backbone, including the EfficientNet-B5 and EVA-02 teachers. Overall, our results show that rigorous control of pretraining and evaluation is essential for interpreting biomedical architecture benchmarks, while distillation may be a more effective route to practical single-cell classification than architecture choice alone.