Global image crops primarily drive semantic learning in vision transformers
Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learning
Computer Vision and Pattern Recognition
Summary
Understanding how self-learning computer vision models understand images is challenging. The authors found that these models learn meaningful image features mainly by comparing two large, different views of the same picture. Smaller local patches only help the model refine what it already knows but don’t create understanding on their own. They also suggest better ways to measure how good the model’s understanding is by looking at how well it matches parts of images, not just at overall classification accuracy. This work helps explain how computers learn to recognize visuals without direct labels.
What this means in practice
- •For computer vision engineers: Improve training strategies for vision transformers by focusing on global image crops to enhance semantic feature learning.
- •For robotics teams: Develop more robust visual perception systems that use global image alignment to better understand and navigate complex scenes.
Authors
Basavaraj Sunagad, Artur Jesslen, Adam Kortylewski
Abstract
Self-supervised vision transformers trained with DINO-style objectives exhibit striking emergent semantic representation quality across visual tasks, yet the mechanisms underlying this behavior remain unclear. We present a systematic empirical dissection of the DINO family and show that semantic representations arise primarily from enforcing consistency between geometrically distinct global views of the same image instance. This instance-specific global alignment acts as the semantic anchor of DINO-style learning. Across controlled retraining experiments evaluated on semantic correspondence and a diverse suite of 2D and 3D downstream tasks, we find that patch-level masking objectives enhance semantics only when trained jointly with this global alignment, indicating that the iBOT objective refines and densifies existing semantic structure rather than creating it independently. In contrast, local-to-global view alignment does not substantially improve semantic qualities at fixed compute beyond a purely global alignment. Beyond training design, we revisit how semantic representation quality should be evaluated: while classification accuracy is the standard validation score, semantic correspondence provides a complementary axis that more reliably predicts downstream task performance. Together, these findings provide a functional decomposition of DINO-style learning and represent an important step toward understanding how semantic representations emerge in self-supervised vision models.