Kernel-based method steers clip embeddings using vision language preferences
Kernel-Based Steering of CLIP with Vision-Language Model Preferences
Computer Vision and Pattern RecognitionMachine Learning
Summary
Comparing images efficiently is hard because big vision-language models give similarity opinions but not simple summaries for fast use. The authors found a way to teach an existing model, CLIP, to match these opinions without needing extra human labels or hidden data. They do this by adjusting CLIP so its image summaries better agree with the bigger models, while keeping its ability to understand images and words together. This method improves how well CLIP finds similar images and keeps its general skills, and it can even control what kinds of differences it learns.
What this means in practice
- •For computer vision engineers: Improve image search systems by updating CLIP embeddings to reflect richer visual similarity judgments without costly new data collection.
- •For multimodal ai developers: Create vision-language models that retain text-image capabilities while learning nuanced visual distinctions from teacher models for better specialized tasks.
Authors
Sajjad Ghiasvand, Haniyeh Ehsani Oskouie, Sina Mansouri, Mahnoosh Alizadeh, Farzan Farnia, Ramtin Pedarsani
Abstract
Large vision-language models (VLMs) can judge visual similarity, but their judgments are not directly available as compact image embeddings for efficient comparison. We study how to transfer these preferences into CLIP while retaining its image--text capabilities. We introduce ASK, a kernel-based steering method that learns from elicited pairwise judgments without accessing teacher embeddings or collecting new human similarity annotations. ASK constructs positive semidefinite target kernels within small image groups and combines visual kernel matching with an image--text distributional anchor. Low-rank adapters jointly update the visual and text encoders while regularizing predictions toward frozen CLIP. After adaptation, retrieval uses CLIP image embeddings and cosine similarity, with no VLM calls. Experiments across five image domains, four CLIP backbones, and six judges evaluate teacher agreement, retrieval, and recognition retention. For ViT-B/16, mean retrieval mAP on classes excluded from adaptation increases from 53.8 to 75.0, compared with 71.7 for DINOv2 targets with KL anchoring. Mean zero-shot accuracy with jointly adapted encoders increases from 61.8\% to 62.4\%, averaged over 12 benchmarks and the five adaptation domains. Prompting provides an additional capability: selecting which visual distinctions the student learns. Human-annotated evaluations across four datasets support this criterion-specific control.