Hyperclip plus plus fine-tunes clip for better open vocabulary segmentation
HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
Computer Vision and Pattern Recognition
Summary
Semantic segmentation means labeling every pixel in an image, which is hard when new classes appear unexpectedly. The authors find that fine-tuning CLIP, a vision-language model, works better when done in hyperbolic space, which helps capture hierarchies. They introduce HyperCLIP++, which adjusts the representation scale to better match detailed pixel-level image data and keeps text and image parts in sync. This approach improves segmentation accuracy while only changing a small fraction of the model’s parameters.
What this means in practice
- •For computer vision developers: Improve open-vocabulary image segmentation by fine-tuning vision-language models more effectively with fewer parameters.
- •For augmented reality creators: Enable more accurate real-time object segmentation by adjusting model embeddings to better capture pixel-level details across different visual categories.
Authors
Zelin Peng, Zhengqin Xu, Changsong Wen, Yu Huang, Yaoming Wang, Xiaokang Yang, Wei Shen
Abstract
CLIP, a foundational vision-language model, has emerged as a powerful tool for open-vocabulary semantic segmentation. While freezing CLIP's text encoder is known to preserve its generalization capability, recent studies show that fine-tuning both CLIP's text and image encoders jointly significantly enhances segmentation performance, especially for classes from open sets. In this work, we explain this phenomenon from the perspective of hierarchy alignment, since during fine-tuning, the hierarchical level of image embeddings shifts from image-level to pixel-level. We achieve this by leveraging hyperbolic space, which naturally encodes hierarchical structures. Our key observation is that, during fine-tuning, the hyperbolic radius of CLIP's text embeddings decreases, facilitating better alignment with the pixel-level granularity of visual data. Building on this, we propose HyperCLIP++, a novel and parameter-efficient adaptation strategy. HyperCLIP++ directly adjusts the hyperbolic radius of CLIP's embeddings via scaling transformations to achieve a hierarchy alignment to the target task, i.e., segmentation. To ensure this hierarchy alignment is effected consistently across both modalities and preserves their cross-modal alignment during training, HyperCLIP++ integrates a Dual Cross-Relation Communication (DCRC) module that synchronizes these adjustments between the vision and text pathways. Our experiments show that HyperCLIP++ achieves state-of-the-art performance across three benchmarks while fine-tuning only approximately 5% of CLIP's total parameters. More importantly, we observe that after adjustment, CLIP's text embeddings exhibit a relatively fixed hyperbolic radius across datasets, suggesting that the hierarchical level required for this segmentation task might be quantified using the hyperbolic radius.