Vision language models learn better by linking images and text transformations
Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models often struggle when trained on new tasks because they treat different transformed images as unrelated, even if they belong to the same class. The authors designed a way to let these models understand image changes by including matching text descriptions of the transformations, helping the models learn features more effectively. Their new method also adjusts the way similarities between images and texts are measured to avoid confusing signals during training. This improves how well the models work across different tasks such as transfer learning and few-shot learning.
What this means in practice
- •For computer vision engineers: Improve image understanding models by conditioning on transformations paired with text to enhance transfer learning accuracy.
- •For machine learning platform teams: Incorporate re-calibrated loss functions to better handle multiple positive examples in vision-language pipelines for robust feature learning.
Authors
Seungmin Oh, Seunghun Kang, Jongbin Ryu
Abstract
Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.