Detecting subtle differences between image datasets with optimized concepts
Detecting Fine-Grained Differences between Image Datasets
Computer Vision and Pattern Recognition
Summary
Telling how two big collections of images are different can be tricky, especially when the differences are small. This paper looks at ways to find which ideas or features make two image sets distinct by picking and improving specific concepts rather than relying on costly language models. The authors created a new test to better spot hard-to-see and combined differences and showed how choosing the most useful and least repetitive concepts helps. Their method also helps find biases or mistakes hidden in datasets, making the process more clear and efficient.
What this means in practice
- •For machine learning engineers: Improve training dataset quality by identifying subtle and low-prevalence differences to reduce biases and spurious correlations.
- •For generative model developers: Analyze outputs from different generative models or prompt settings to detect fine-grained conceptual differences for better model refinement.
Authors
Leander Girrbach, Alexander von Recum, Kirill Bykov, Zeynep Akata
Abstract
Detecting whether and in particular how two data distributions differ is a fundamental statistical problem. For example, we might be interested in which concepts (expressed in natural language) differ between two classes, the outputs of different generative models, or the same model with different prompt specifications. When working with large and complex image datasets, we intuitively think that first selecting concepts from a broad list using fine-grained patch embeddings, and then optimizing the concepts to better fit the data is an effective workflow. However, prior work instead uses MLLMs to propose concepts and image-level embeddings to select them. Therefore, we systematically compare these techniques and find that selecting concepts from broad lists can replace MLLM-proposals at obviously lower cost. MLLM inference can then be spent on optimizing selected concepts instead to achieve substantial improvements. Additionally, by designing a new benchmark that focuses on low-prevalence and multi-concept distinctions, we find that selecting nonredundant concepts instead of only the most distinguishing concepts matters, and we introduce techniques for how to effectively do this. Finally, we show how our methods address important problems like detecting spurious correlations and biases. Overall, our work presents a fundamental perspective shift on how we analyze interpretable distributional differences between datasets.