GDBA-Net improves nuclei segmentation across different tissue samples

GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation

Computer Vision and Pattern Recognition

Summary

Segmenting cell nuclei in tissue images is important but tricky because tissues vary a lot and images can look very different. The authors developed GDBA-Net, a lightweight AI model that combines two types of information—semantic knowledge and structural details—to better identify nuclei even when the image data changes a lot. It adapts a preexisting model called SAM2 without retraining it entirely, using only a small number of parameters. Their tests show it outperforms other methods on different datasets, making it more reliable for medical image analysis.

What this means in practice

  • For medical image software developers: Create pathology image analysis tools that maintain high accuracy when analyzing tissue images from different labs with varying staining and preparation.
  • For digital pathology service providers: Offer more robust nuclei segmentation as part of digital pathology workflows that can adapt quickly to varied histopathology image domains with minimal retraining.$Commercial implications: Enables scalable services for diverse clinical labs by adapting segmentation to multiple imaging variations, improving diagnostic software's market fit.

Authors

Biplab Poudel, Betserai Albert Magarire, Anish Kumar Shah, Jiacheng Xie, Dong Xu

Abstract

Accurate nuclei segmentation underpins quantitative computational pathology, yet remains challenging due to morphological variability, weak boundaries, nuclear crowding, and pronounced domain shifts arising from differences in tissue preparation, staining, and acquisition. While deep learning and recent foundation models have improved segmentation performance under matched distributions with the original training data, their generalization across heterogeneous histopathology domains remains limited. We introduce GDBA-Net, a parameter-efficient nuclei segmentation framework built around the proposed Gated Dual-Branch Adapter (GDBA). GDBA-Net adapts a frozen SAM2 encoder by integrating pretrained semantic representations with explicit frequency-derived structural priors. The complementary representations are independently transformed through Grid MLP branches and dynamically integrated using sample-adaptive channel-wise gating, selectively emphasizing structural cues relevant to nuclear delineation. Despite using only 4.3 million trainable parameters, GDBA-Net achieves state-of-the-art in-dataset performance and consistently superior cross-dataset generalization across six diverse nuclei segmentation benchmarks under fully supervised, cross-dataset, and five-shot adaptation protocols, outperforming competing CNN-, transformer-, and foundation model based methods. Comprehensive ablations confirm that these gains arise from the coordinated design of semantic-frequency complementarity and adaptive gating rather than increased model capacity. By uniting transferable semantic representations with explicit structural priors, GDBA-Net provides a robust, annotation-efficient solution to domain variability in nuclei segmentation. https://github.com/biplabpoudel25/GDBA_Net