Temperature modulation improves multimodal contrastive learning accuracy
TeMo: Temperature Modulation for Multimodal Contrastive Learning
Computer Vision and Pattern Recognition
Summary
Contrastive learning helps computers tell apart things that are similar and different, using a setting called temperature to control how much difference matters. Most methods keep this temperature fixed or use one value for everything, but the authors introduce TeMo, which changes the temperature based on how similar each pair is. This lets the model learn more detailed relationships between different kinds of data, like images and text. Their experiments show that this approach improves performance on tasks like recognizing images without prior training examples.
What this means in practice
- •For machine learning engineers: Enhance image-text retrieval systems by adaptively adjusting training parameters for better cross-modal similarity learning.
- •For natural language processing teams: Improve multimodal classifiers that combine text and images by capturing different semantic granularities during training.
Authors
Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva
Abstract
Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter $τ$, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.