Multi resolution approach improves zero shot image segmentation accuracy
MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation
Computer Vision and Pattern Recognition
Summary
Zero-shot image segmentation means identifying objects in pictures without having seen examples of those objects before. The authors found that existing methods like CLIP struggle with this task because they lose detail and structure when analyzing images at a low resolution. To fix this, they created MARS-CLIP, which combines detailed local features with a bigger picture view and uses attention techniques to better outline object edges. Their tests show that this method works better than previous ones on several different image sets.
zero-shot learningimage segmentationCLIPmulti-resolutionattention mechanismself-attentionsemantic segmentationfeature extractionobject boundariescontrastive learning
Authors
Nagito Saito, Shintaro Ito, Koichi Ito, Takafumi Aoki
Abstract
Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and Attention Refined Segmentation for CLIP), a novel framework for zero-shot semantic segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-attention block to accurately restore object boundaries. A set of experiments on six public datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.