The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
2026-07-20 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors found that people judge how similar images are based on different qualities like shape or color, but existing tools only give one overall similarity number. They created a big dataset where people compared sets of three images with notes on different aspects of similarity. Using this, they trained a vision-language model to make a new similarity measure called TPIPS, which can focus on specific features when given a text prompt. This new method matches human judgments better and works well even on new images. It can also improve tasks like searching with text or evaluating generated images.
human visual similarityperceptual similarity metricsvision-language modelsimage tripletstext-prompted similarityfine-tuninggenerative models evaluationcompositional searchimage retrieval
Authors
Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, Alexei A. Efros, Richard Zhang
Abstract
Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS