Pretrained visual models reveal meaningful image parts at multiple scales
Discovering Semantic Granularity in Images through Clustering of Pretrained Visual Representations
Computer Vision and Pattern Recognition
Summary
Images have different meaningful parts, from big objects to smaller details, called semantic granularity. This paper studies if existing pretrained visual models already know these levels without being told explicitly. The authors found a way to group image pixels using those models' features into meaningful segments, from broad to fine. They also created measures to tell how good these groupings are and showed that their method works well on popular image datasets.
What this means in practice
- •For computer vision engineers: Automatically generate meaningful image segmentations at multiple detail levels using pretrained models without manual labeling.
- •For digital content creators: Enable content editing tools to isolate and manipulate meaningful image regions discovered from pretrained model features without custom training.
Authors
Ying Dai
Abstract
Semantic granularity, which refers to the levels at which visual representations organize image content into meaningful entities, is fundamental to image understanding. However, whether pretrained visual representations inherently encode semantic structures at multiple granularities remains poorly understood. This paper presents a representation-driven approach for discovering semantically meaningful image regions at different granularities through clustering of pretrained visual representations without semantic supervision. Specifically, we concatenate selected feature maps extracted from a pretrained visual model and apply a singular value decomposition (SVD)-based projection to derive pixel-wise principal representation maps (PRMs). The resulting PRMs capture implicit semantic structures embedded in pretrained visual representations, making them accessible to subsequent semantic clustering. We then introduce the semantic continuity rate (SCR), based on the silhouette principle, to quantify the coherence of pixel-wise representations within and between clusters at different candidate granularities. Furthermore, we analyze the signal-to-noise ratio (SNR) of SCR profiles to assess representation suitability for discovering semantic regions at multiple granularities. Our results indicate that PRMs exhibiting higher SNR values over an initial range of cluster numbers tend to yield more semantically meaningful regions across multiple granularities. We evaluate the proposed approach on the COCO-2017 and VOC-2012 validation datasets using Hungarian-matched intersection over union (HIoU). The results demonstrate competitive segmentation performance among the compared methods, supporting the feasibility of discovering useful semantic granularities through clustering PRMs without semantic supervision.