Pretrained vision transformers match CNNs on thin retinal vessels
Evaluating SAM-based and Convolutional Architectures in Thin Retinal Vessel Segmentation: Model Capacity, Computational Cost and Foundation Pre-training
Computer Vision and Pattern Recognition
Summary
Segmenting thin blood vessels in images of the retina helps doctors see eye problems. People thought Vision Transformer models would find these thin vessels worse than traditional convolutional models because of how they analyze images. The authors compared several models including transformers and CNNs on multiple datasets. They found that pretrained transformers performed about as well as the best CNNs at detecting thin vessels. However, smaller models can also achieve similar accuracy.
What this means in practice
- •For medical imaging engineers: Design retinal vessel segmentation systems using pretrained transformers to match CNN accuracy on thin vessels while considering model size and training requirements.
- •For microscopy image analysts: Improve thin vessel detection in microscopy datasets by selecting from pretrained transformer or CNN models with comparable performance.
Authors
Carlos Eduardo Fontaneli, Cesar Henrique Comin
Abstract
In retinal vessel segmentation, plain Vision Transformers (ViTs) are commonly expected to perform worse than convolutional neural networks (CNNs) on thin vessels, for reasons such as their coarse patch tokenization, their weak locality bias and their need for large training sets. Although this expectation is widely shared in the literature, how plain ViTs actually compare with CNNs on thin vessels has not been systematically quantified. We address this question by comparing a model with a plain ViT encoder (MedSAM), a hybrid Transformer (LiteMedSAM) and three CNNs of different sizes on three retinal datasets (DRIVE, HRF, FIVES) and, as a second-domain check, a microscopy dataset (VessMap). We measured the centerline recall and precision of thin vessels and compared the models while also considering their computational cost and how many epochs they needed to train. We found that the pre-trained MedSAM had a centerline recall of thin vessels that was not significantly different from that of the best CNN on all four datasets (-0.6 to +2.5 percentage points). A compact plain ViT obtained through architecture search but trained from scratch did not retain this performance. For pre-trained plain ViTs, our results do not support the expectation that they miss thin vessels, although similar thin-vessel performance can be obtained with much smaller models.