Temperature modulation improves multimodal contrastive learning performance
TeMo: Temperature Modulation for Multimodal Contrastive Learning
Computer Vision and Pattern Recognition
Summary
Contrastive learning helps computers recognize related and unrelated items by comparing them. A key setting called temperature controls how strongly the model treats differences between unrelated pairs. The authors introduced TeMo, a method that adjusts this temperature dynamically for each pair based on how similar they are, making the learning more precise. This approach helps the model understand both broad and detailed connections during training. Their experiments show that TeMo consistently boosts performance in tasks like image and text matching without needing special tweaks.
contrastive learningtemperature hyperparametermultimodal learningsimilarity measurenegative samplespositive sampleszero-shot retrievalclassificationtemperature modulationfine-grained semantics
Authors
Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva
Abstract
Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter $τ$, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.