Contrastive loss tuning improves clip post-training without breaking compatibility
Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Fine-tuning the vision part of CLIP, a model that links images and text, can sometimes cause it to forget what it learned. The authors found that the usual way to measure differences during this fine-tuning was set incorrectly, causing problems with small batches. By adjusting this setting and keeping the text part unchanged, they created a method called ComCLIP that improves the vision model’s performance without causing issues for other systems that use it. This means the refined model can be swapped in easily and works better for new image tasks without hurting old ones.
What this means in practice
- •For computer vision engineers: Improve pretrained vision models by applying better contrastive tuning during post-training without needing to retrain text encoders.
- •For multimodal ai developers: Enhance visual feature transferability in vision-language systems by swapping in refined vision encoders that maintain downstream compatibility.
Authors
Zidan Wang, Yaqian Li, Xiaokai Zhang, Kaiwen Long, Kun He, Hanpeng Liu
Abstract
CLIP serves as a foundational vision-language model and the de facto vision encoder for downstream VLMs such as LLaVA. Post-training offers a lightweight route to refine CLIP, but recent work argues that the standard contrastive loss is unsuitable for post-training due to catastrophic forgetting under small batches, motivating designs that abandon the contrastive objective in favor of distillation. We revisit this premise and find that, for the InfoNCE objective, the reported forgetting is driven primarily not by insufficient negatives but by an inappropriate magnitude of the contrastive temperature $τ$: with $τ$ set sufficiently small, contrastive post-training improves rather than degrades the pretrained CLIP, which we explain through the temperature dependence of the InfoNCE gradient. Building on this finding, we propose \textbf{ComCLIP}, a lightweight single-epoch post-training recipe that freezes CLIP's text encoder---so the refined vision encoder is a drop-in replacement with unchanged architecture and inference cost---and trains the vision encoder with a properly-tempered contrastive loss, an MSE anchoring loss against the original CLIP, and a relational distillation loss from DINOv2. Over multiple seeds, ComCLIP matches the self-distillation baseline CLIP-Refine on zero-shot classification while significantly improving the transferability of visual features, measured by linear probing ($48.99$ vs.\ $42.28$ on ViT-B/16), and on ViT-L/14 it also improves MMVP over CLIP-Refine ($24.20$ vs.\ $19.01$); CLIP-Refine remains stronger on image-text retrieval. Used as a drop-in vision encoder for LLaVA-1.5-7B without re-aligning the projector or LLM, ComCLIP yields no net change across $8$ VLM benchmarks, i.e., the refinement does not break downstream compatibility. Code and models are available at https://github.com/showstarpro/ComCLIP.git.