Mars clip improves zero shot image segmentation accuracy
MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation
Computer Vision and Pattern Recognition
Summary
Image segmentation means breaking an image into meaningful parts, like objects, without training on those exact images beforehand. The authors show that a popular AI model called CLIP struggles to do this well because it loses detail and context in the image. They designed a new system, MARS-CLIP, that looks at images at different sizes and pays special attention to spatial details and colors. This helps the system better recognize and outline objects it hasn't seen before, improving performance on several tests.
What this means in practice
- •For computer vision engineers: Create zero-shot image segmentation tools for varied datasets without needing new labeled data.
- •For robotics software developers: Enable robots to better identify and separate objects in unknown environments using visual input.
Authors
Nagito Saito, Shintaro Ito, Koichi Ito, Takafumi Aoki
Abstract
Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and Attention Refined Segmentation for CLIP), a novel framework for zero-shot semantic segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-attention block to accurately restore object boundaries. A set of experiments on six public datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.