Multimodal large language models improve few-shot image segmentation accuracy
Exploiting Target Knowledge from MLLMs for Robust Few-Shot Segmentation
Computer Vision and Pattern Recognition
Summary
Segmenting new types of objects in images with only a few examples can be hard, especially if the objects look very different or partly covered. The authors propose using advanced language models that understand images and text to get extra clues about where objects might be and what they are. They combine this spatial and semantic information to help the computer find the target better in the image. Their approach shows better results than previous methods on segmenting new objects from very few examples.
What this means in practice
- •For computer vision engineers: Develop more accurate few-shot object segmentation models by integrating spatial and semantic knowledge from multimodal language models.
- •For medical imaging specialists: Improve segmentation of rare or unseen anatomical structures in scans using few labeled examples enhanced with multimodal model reasoning.
Authors
Yijun Hu, Heng Fan, Libo Zhang
Abstract
Few-shot segmentation (FSS) aims to segment unseen object categories with a few (e.g., one or five) labeled examples, enabling efficient adaptation to novel classes. Conventional models typically rely on appearance-based visual matching between support and query images for segmentation. While straightforward, these methods often struggle to handle significant appearance discrepancies and occlusions in the query image due to insufficient target knowledge. To mitigate this, we introduce a novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS. Specifically, building on SAM 2, our method, named MK-FSS, exploits two forms of complementary knowledge derived from a query image by an MLLM for FSS, including spatial knowledge, which provides a spatial prior indicating the potential target location, and semantic knowledge, which describes the target using text. The spatial knowledge is first encoded into a memory representation, and then resulting memory is integrated with the support-guided memory feature from query image through a carefully designed dual-memory debate-fusion (DMDF) module, yielding a more robust target memory feature. In parallel, the semantic knowledge is encoded into the textual feature, which is fused with multi-scale query features via a progressive cross-modal prompt generator (PCPG), producing a target-aware multimodal prompt for segmentation. Working together, the dual-memory feature and the multimodal prompt provide a comprehensive representation of the target, enabling more robust segmentation. In our extensive experiments, MK-FSS shows promising results and largely surpasses existing methods. Code will be released.