Improving vision language model training with smarter image text matching
Re-calibrated Contrastive Loss for Transformation-Aware Prompt Conditioning in Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Training vision-language AI models can be tricky when images and their descriptions don't line up well during learning. The authors found that treating different views of the same thing as totally separate makes it harder for the AI to learn. They fixed this by letting the model know when images are just transformed versions of the same object, using special text cues for these changes. This helps the AI understand images and text better together, especially when adapting to new tasks or data. Their approach led to better performance in various tests compared to older methods.
vision-language modelstransfer learningcontrastive lossprompt conditioningdata augmentationdistribution shiftfew-shot learningcross-entropy losspositive pairs
Authors
Seungmin Oh, Seunghun Kang, Jongbin Ryu
Abstract
Ensuring effective transfer learning for vision-language models without compromising their generalization performance is crucial. However, many existing methods overlook data characteristics and simply reuse the training strategies adopted during pre-training. Specifically, they treat same-class samples as distinct instances and transform images independently of their paired text prompts, which makes model learning more difficult. We address these limitations through transformation-aware prompt conditioning and a re-calibrated contrastive loss. Fixed text descriptors identify the transformations applied to paired images, providing transformation-level consistency without altering class semantics. This design aligns the image and text branches at the transformation level, enabling richer representations while preserving the models' ability to generalize. In addition, our loss function mitigates positive-gradient dilution in soft-target cross-entropy when each anchor has multiple valid positives. During transfer, our approach treats same-class samples as positives rather than distinct instances, enabling the model to learn domain-specific features more effectively. Experiments across distribution shift, transfer learning, and few-shot settings demonstrate consistent improvements over existing approaches. Source code for our method is available at https://github.com/SoongE/ReCalCon.