AI summaryⓘ
The authors created SOCO, a new test to check how well computer vision models can recognize and match specific parts of objects across different images and categories. They included detailed labels for over a million pairs of parts in 100 object categories, with descriptions in language to also test models that understand images and text together. Their experiments showed that while standard vision models understand object parts well, they struggle to match parts across similar categories and don't fully capture part positions. Models that combine vision and language perform better at locating parts from text hints but aren't as good at matching parts visually across images. The study also found that how well a model matches parts predicts its success on other detailed tasks better than traditional image recognition scores.
semantic correspondencekeypoint annotationvision foundation modelslarge vision-language modelsobject partsmultimodal modelspart-level understandingbenchmarkimage segmentation3D pose estimation
Authors
Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski
Abstract
Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint language descriptions, enabling the evaluation of large vision-language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i) vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts performance on dense downstream tasks, including segmentation, tracking, 3D pose estimation, and 3D detection, more strongly than ImageNet classification. Together, these findings position SOCO as a benchmark for structured, part-level representation quality in vision and multimodal foundation models.