Visual memory boosts segmentation accuracy with text alignment
Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation
Computer Vision and Pattern Recognition
Summary
Few-shot segmentation tries to spot specific objects in images while learning from only a few examples. The authors built on a tool called SAM 3 that understands images and text but struggles with specialized subjects. They combined a visual memory that remembers example features and masks with text features, making the system better at matching what it sees with the words used to describe it. This approach improved accuracy by over 5% compared to previous methods.
What this means in practice
- •For medical imaging teams: Improve segmentation of rare or specialized medical objects using few annotated examples and combined visual and text features.
- •For robotics developers: Create robots that recognize and segment unusual objects by integrating example visuals with language descriptions despite limited data.
Authors
Tianqi Wei, Zhi Chen, Xin Yu, Scott Chapman, Zi Huang
Abstract
Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance remains limited in specialized domains, where semantic prompts and visual representations are often poorly aligned. To address this challenge, we propose ViTA-SAM, a framework built upon SAM 3 that integrates visual memory and textual features through cross-modal alignment. Specifically, we construct a visual memory module that encodes support features together with their corresponding masks, providing support-aware visual guidance for query prediction. In parallel, we leverage the text encoder of SAM 3 to extract semantic features as complementary priors. To bridge the gap between semantic and visual representations, we introduce a cross-modal alignment module that conditions textual features on memory-enhanced visual representations, yielding more consistent and visually grounded predictions. In addition, we propose a pseudo-label refinement strategy that updates the memory with predicted query masks, enabling query-adaptive representation learning. Extensive experiments on multiple benchmarks show that ViTA-SAM achieves the best overall performance among the compared methods, outperforming the strongest competing method by over 5% in average mIoU.