Masked patch self-distillation balances semantic meaning and local image detail
The Locality Trade-off in Masked Patch Self-Distillation
Computer Vision and Pattern Recognition
Summary
This paper looks at how a technique called masked patch self-distillation, used in computer vision models, affects understanding small details versus bigger picture meaning in images. The authors find that while this method improves recognizing overall scenes or objects, it also makes it harder to track exact local patterns or textures, especially in smaller models. Larger models handle this trade-off differently, still improving spatial matching but making pixel-level details less accessible. The findings suggest that future visual AI systems need to carefully balance learning semantic concepts, preserving fine details, and maintaining spatial relationships in images.
What this means in practice
- •For computer vision engineers: Improve model design by balancing semantic understanding with local detail preservation in vision AI systems.
- •For image recognition developers: Develop more robust image recognition systems that consider both global semantics and spatial details using insights on locality trade-offs.
Authors
Xi Weng, Zhaoyu Liu, Lianyu Hu, Jin Song Dong
Abstract
Masked patch self-distillation is a key component of modern vision foundation models, including DINOv2 and DINOv3, yet its implications for local visual information remain unclear. We show that its semantic benefits can coexist with a scale-dependent locality cost. Using two annotation-free diagnostics, we distinguish recoverable local appearance from spatial correspondence. In smaller models, adding iBOT substantially weakens patch correspondence while largely preserving appearance decodability. In larger models, correspondence improves, but pixel and texture evidence becomes less linearly accessible in deep features. Thus, scaling changes the form of the locality cost rather than uniformly eliminating it. Our analysis of Gram anchoring further shows that preserving patch relations does not guarantee simultaneous improvements in both properties. These findings motivate evaluating and designing visual representations around the complementary demands of semantic abstraction, appearance preservation, and spatial correspondence. Our code is available at https://github.com/winci-ai/jea.