Feature matching method improves compact visual model training accuracy
The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Training smaller AI image models to copy bigger ones often focuses too much on easy-to-match parts, missing important subtle details. The authors found that this common training method favors dominant parts of the representations and neglects less obvious, but crucial information. They designed a new training method called SpecMatch that pays more balanced attention to all parts of the model’s knowledge. Tests show SpecMatch helps smaller models perform better in many areas, including image recognition and protein analysis.
What this means in practice
- •For computer vision engineers: Train smaller visual models to better mimic large foundation models by improving how their features are matched during knowledge distillation.
- •For bioinformatics developers: Enhance protein understanding models by applying balanced feature matching to improve downstream task performance.
Authors
Kuniaki Saito, Yoshitaka Ushiku
Abstract
Large visual foundation models have demonstrated remarkable transferability across a wide range of downstream tasks. To deploy such models efficiently, feature matching has become a popular knowledge distillation approach that transfers teacher representations to smaller student models without requiring labeled data. However, we show that the conventional feature matching objective with L2-distance is inherently biased toward reconstructing dominant spectral directions of the teacher representation, while under-optimizing low-variance directions that often contain task-relevant information. To address this, we propose Spectrum-Balanced Feature Matching, SpecMatch, a simple objective that adaptively emphasizes under-optimized spectral directions while preserving the relative importance of dominant directions. SpecMatch is easy to implement and introduces negligible computational overhead. Extensive experiments on image recognition demonstrate that SpecMatch consistently improves downstream adaptation across diverse tasks, including image classification, anomaly detection, medical image analysis, and domain generalization. In particular, SpecMatch outperforms conventional feature matching in 40 of 42 teacher--student and training-setting combinations, while consistently improving over the original student model in all settings. We further demonstrate that the proposed objective generalizes beyond vision, improving downstream performance across six protein understanding tasks.