Repurposing CLIP to Localize at Pixel Level
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors present CLIPix, a method that turns the image-level understanding of the CLIP model into detailed pixel-level maps showing where objects are in an image. They do this by following how CLIP makes decisions to find attentive regions tied to specific objects and improve these signals to reduce errors caused by global image biases. They also combine localization with detailed features to get accurate segmentations. Tests on standard datasets show that their approach works well for identifying and segmenting various objects in images.
CLIPVision-Language Modelspixel-level localizationdense predictionimage segmentationobject localizationglobal feature biasPASCAL datasetCOCO datasetattentive regions
Authors
Jiaxiang Fang, Shiqiang Ma, Jing Wang, Siyu Chen, Fei Guo, Shengfeng He
Abstract
Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases. In this paper, we introduce CLIPix, a simple yet effective framework that repurposes CLIP to perform pixel-level localization. By tracing back CLIP's classification process, CLIPix identifies object-specific attentive regions and repurposes them as pixel-level localization cues. To address noise introduced by global biases, we propose a Noise-Resistant Correction strategy, refining these cues for more precise segmentation. Additionally, we introduce a Localization Embedding strategy to integrate both localization and enriched detail information, enabling accurate, high-resolution segmentation. Our approach preserves CLIP's generalization strength and unlocks its potential for segmenting arbitrary objects. Extensive experiments on the PASCAL and COCO datasets demonstrate that CLIPix achieves state-of-the-art performance, underscoring its effectiveness.