Self supervised vision model learns interpretable image parts without labels
DisParQ: Self-Supervised Part Concepts for Interpretable Vision Foundation Models
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Understanding how AI models see images is tricky because their internal features are often hidden. The authors created DisParQ, a method where the model learns to represent images using meaningful parts and their variations without needing any labels or language help. This lets us see which parts the model uses to recognize things and even find similar parts across different objects. DisParQ works well on a variety of image tasks and matches the performance of top AI models.
What this means in practice
- •For computer vision engineers: Create interpretable image recognition systems that reveal what parts influence model decisions without requiring labeled data.
- •For digital asset managers: Enable search and retrieval of images by matching object parts across different categories to improve content organization and discovery.
Authors
Adam Pardyl, Siddhartha Gairola, Sukrut Rao, Adam Wróbel, Bartosz Zieliński, Bernt Schiele, Dawid Rymarczyk
Abstract
Concept-based vision models represent images through an intermediate layer of human-inspectable concepts, so what a model relies on can be traced to those concepts. However, those models are often limited to fixed categories or depend on language to define their concepts. We introduce DisParQ (Discrete Parts with Quantized attributes), a method that learns spatially grounded, discrete concept representations from a powerful frozen vision-only self-supervised backbone. It requires no class labels and no language supervision. Each image patch is assigned to exactly one concept from a learnable prototype dictionary, and only a sparse subset of concepts may activate per image. To capture how each concept varies across images (e.g., the type of a "wheel"), we learn continuous residuals alongside the concepts and then quantize them into discrete attributes. A spatial decoder reconstructs the backbone's representation from the concepts and attributes alone, so successful reconstruction means that the discrete representation preserves the backbone's information. We evaluate DisParQ across seven datasets, from general recognition (ImageNet, PartImageNet, Places) to fine-grained benchmarks (CUB, Cars, Dogs, Flowers). We show that DisParQ closely matches its frozen DINOv2 teacher on ImageNet linear probing (83.2% top-1), achieves higher concept consistency than language-aligned models, remains competitive on fine-grained recognition, and enables cross-category part-based retrieval.