Semantic spatial residual banks improve multi mask medical image segmentation

Single-Annotation, Multi-Hypothesis Segmentation with Semantic-Spatial Residual Banks

Computer Vision and Pattern RecognitionMachine Learning

Summary

Some images can be segmented in different valid ways, but training data usually only shows one example. The authors study how to train models that, given one example mask per image, can produce multiple masks that cover other possible human annotations. They find that combining fixed semantic features with learned image residuals helps models capture both overall structure and local details in segmentations. Their new method, Semantic-Spatial Residual Bank, performs better on medical images and natural images compared to previous approaches when multiple masks are needed.

What this means in practice

  • For medical image analysis teams: Generate multiple plausible segmentations from limited annotations for more comprehensive medical image evaluation.
  • For computer vision engineers: Improve segmentation models to represent structural and local variations in ambiguous images using semantic and image residual features.

Authors

Armin Dadras, Robert Sablatnig, Franziska Proksa, Markus Seidl

Abstract

Multi-rater segmentation datasets show that a single image can admit several valid masks, yet most training sets expose only one annotation per image. We study single-annotation multi-hypothesis segmentation: each training image exposes one fixed mask, while the model must return a finite bank that covers held-out human annotations at test time. Our experiments reveal a representation tradeoff. Banks learned directly from images recover variations in local boundaries and extent but lose structural hypotheses, whereas frozen semantic features preserve structural grouping but underrepresent spatial variation. We introduce the Semantic-Spatial Residual Bank (SSRB), which freezes a semantic bank trained with balanced assignments and learns a residual from the image for each hypothesis. Across five medical endpoints, SSRB improves coverage of held-out masks at $K=8$ over a Probabilistic U-Net under the same one-mask constraint. On LIDC and LIDC-hard, it also produces substantially more distinct useful masks. On BSDS, it preserves the semantic bank's performance in grouping regions. Its advantage grows as more masks are returned, showing that residuals learned for corresponding heads improve coverage of local annotation variation while preserving how regions are grouped.