Vision transformers learn where to look in pathology images
Learning Where to Focus: Self-Supervised Multi-Scale ViTs for Histopathology
Computer Vision and Pattern Recognition
Summary
Pathologists examine tissue samples by first spotting suspicious areas and then zooming in for a closer look. The authors found that existing computer models evenly analyze all parts of an image, ignoring that important clues are spread unevenly. They created a new method called CRAFT that teaches the model to focus more on interesting regions while keeping a broad overview. This approach improves disease diagnosis tasks and uses less computing power, working well even with smaller datasets and models.
What this means in practice
- •For medical imaging teams: Improve computer-aided diagnosis by focusing analysis on suspicious tissue areas with less computation.
- •For digital pathology platform developers: Integrate adaptive multi-scale image analysis to enhance pathology model performance while reducing resource use.$Commercial implications: This approach enables more efficient pathology AI products that require smaller models and less computing power.
Authors
Anabel Stammer, Valay Bundele, Mehran Hosseinzadeh, Hendrik P. A. Lensch
Abstract
Pathologists diagnose diseases by first locating suspicious tissue and then examining it at higher magnification, whereas self-supervised vision transformers (ViTs) allocate the same spatial resolution to every image region despite diagnostic evidence being sparse and spanning multiple biological scales. Recent pathology foundation models have substantially improved representation quality by scaling training data and model capacity, but largely retain uniform tokenization. We instead investigate whether pathology representations can be improved by learning where to allocate spatial resolution during self-supervised learning. To this end, we propose CRAFT (Coarse-to-fine Region-Adaptive Feature Tokenization), a DINO-based framework that learns image-dependent mixed-scale representations by using self-supervised attention to selectively refine informative regions while preserving coarse context, together with a symmetric cross-scale regularization objective that encourages complementary coarse and fine representations. Across CAMELYON16, TCGA-Lung subtype classification, and TCGA-LUAD survival prediction, CRAFT consistently outperforms comparable-scale self-supervised methods while requiring lower inference computation. Despite using only a compact 22M parameter backbone trained on comparatively small pathology datasets, CRAFT remains competitive with, and often surpasses, substantially larger pathology foundation models.