Histology models need less geometric test-time augmentation over time

When Do Histology Foundation Models Need Geometric Test-Time Augmentation?

Computer Vision and Pattern RecognitionMachine Learning

Summary

When looking at tissue images, computers often rotate and flip images multiple ways at test time to improve predictions, because these images don’t have a standard orientation. The authors studied whether this trick is still needed for newer, self-supervised models that already understand rotation well. They found that although rotating and flipping images still helps, the improvement is much smaller for these modern models and especially tiny for the biggest ones. They also showed that running these rotations only on uncertain cases can save time while keeping most of the advantage. However, this approach didn’t help when analyzing whole slide images using a common method called multiple instance learning.

What this means in practice

Authors

Lars Chen, Valentino F. Foit, Dorry Segev, Michal Mankowski

Abstract

Test-time augmentation (TTA) aggregates model predictions across multiple transformed views of an input during inference. Because histology images lack a canonical orientation, $D_4$ TTA---pooling across the 8 rotations and reflections of a square---is a standard default in computational pathology. However, the rise of self-supervised histology foundation models (FMs) with inherent rotational invariance calls its continued utility into question. We benchmark 9 histology FMs and 6 general-purpose models across four patch-level and two whole-slide image (WSI) datasets. We find that $D_4$ TTA remains a reliable post hoc improvement, but benefits diminish from $+1.98$ percentage points (pp) for general-purpose backbones to $+0.50$ pp for histology FMs on TCGA-UT, and shrink further for the largest models. To eliminate computational overhead, our experiments demonstrate that entropy routing---applying TTA selectively to uncertain predictions---recovers most of the gain while augmenting only ${\sim}13\%$ of samples. Finally, we show that patch-level TTA improvements do not transfer to slide-level multiple instance learning (MIL). Code: https://github.com/lars-chen/histology-tta