Semantic visual memory improves few shot image segmentation accuracy

MESAM: Enhancing SAM2 for Few-Shot Segmentation through Semantic-Visual Memory Generation and Refinement

Computer Vision and Pattern Recognition

Summary

Segmenting objects in an image with only a few examples is hard because things can look very different between pictures. The authors studied how to use a tool called SAM2 better by creating a memory that combines both the meaning of an object and its visual details. They made a new system called MESAM that builds and refines this memory to help find objects more precisely, even with big appearance differences. Their experiments show MESAM works better than other methods on several standard image datasets.

What this means in practice

  • For computer vision engineers: Build image segmentation tools that can identify new object classes from very few examples using semantic-visual memory enhancements.
  • For medical image analysts: Improve segmentation accuracy in cases with limited annotated medical images by incorporating semantic and visual memory refinement methods.

Authors

Tianyu Zou, Shengwu Xiong, Rui Wu, Ruilin Yao, Yi Rong

Abstract

This paper investigates how to leverage the powerful capabilities of the Segment Anything Model 2 (SAM2) to tackle few-shot segmentation (FSS) tasks. One major challenge lies in the substantial appearance differences between the support and query images, which may diminish the effectiveness of the memory-matching process in SAM2. Through systematic analysis, we find that the memory generated based on the categorical semantic information is robust to this appearance divergence and is more compatible with SAM2. Based on these observations, we propose a new Memory-Enhanced SAM2 (MESAM) method, which first introduces a Semantic-Visual Memory Generation (SVMG) module to separately construct a query semantic memory, a query foreground visual memory, and a support adaptive visual memory. Then, in our designed Interactive Semantic Memory Refinement (ISMR) module, we further enrich the obtained semantic memory with fine-grained visual details, and suppress its misatcivated background responses. Extensive experiments on the PASCAL-$5^i$, COCO-$20^i$, and LVIS-$92^i$ datasets demonstrate that MESAM consistently outperforms existing FSS methods, achieving new state-of-the-art segmentation performance. Our source code is available at: \href{https://github.com/tianyu-zou/MESAM}{https://github.com/tianyu-zou/MESAM}.