Vision language models match better using rank based comparison
RankAlign: Vision-Language Representation Alignment via Rank Transformation
Computer Vision and Pattern Recognition
Summary
Matching categories between images and words without direct pairing is tough because the internal ways models represent similarity differ. The authors propose a method called RankAlign that changes similarity scores into ranks to make comparisons fairer across these different ways. This approach helps find correspondences between vision and language classes more reliably, even if the data labels are noisy or come from different sources. Their method works across various models and datasets, improving alignment without needing extra pairing information.
What this means in practice
- •For machine learning engineers: Improve matching and transfer between vision and language models using rank-based similarity alignment without cross-modal pairing data.
- •For data integration teams: Match categories from unpaired image and text datasets more robustly even under noisy labels or domain changes.
Authors
Enzhe Zhao, Marco Fiorucci, Lamberto Ballan
Abstract
While the Platonic Representation Hypothesis posits that vision and language models converge toward compatible latent geometries, recovering class-level correspondences between unpaired modalities without cross-modal supervision remains challenging. Existing relational matching methods compare within-modality similarity kernels directly, although the same semantic structure may be represented through different numerical parameterizations across modalities. We propose RankAlign, a rank-based normalization strategy for class-level unpaired vision-language matching. RankAlign transforms each within-modality similarity kernel into an ordinal relational representation before permutation matching, while leaving the correspondence space and matching solver unchanged. The resulting representation is invariant to strictly increasing reparameterizations of similarity values while preserving the ordinal structure used for matching. We provide exact-recovery and stability guarantees that characterize correspondence recovery under both exact and partial ordinal agreement. Experiments across diverse frozen vision and language encoders and multiple datasets demonstrate improved and more stable class-level correspondence recovery, including under label corruption and domain shift.