Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision
Computer Vision and Pattern Recognition
Summary
The authors studied how well deep neural networks (DNNs) match human mental images of objects, which are usually measured by how similar people think objects look. They used a special math tool (GWOT) to check if the networks aligned with humans at a detailed level, even for new objects not used in training. By fine-tuning DNNs with a method called Relational Knowledge Distillation, they made the networks' overall sense of object similarities more like humans', especially in how groups of objects are arranged. However, the closest matches between individual objects didn't improve much. This shows that transferring human-like relationships can help DNNs organize knowledge like humans do, without direct supervision.
Authors
Yuria Shimizu, Soh Takahashi, Takato Horii, Masafumi Oizumi
Abstract
Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently similar to human mental representations, which are not directly observable and are therefore commonly measured through large-scale similarity judgments of object images. A natural approach to narrowing this gap is to directly transfer the relational structure of human representations into DNNs, and previous studies have reported improved human-DNN representational similarity. However, whether this improvement holds under stricter evaluation remains untested in two respects: fine-grained alignment at the individual-object level, and generalization to a human embedding derived from a dataset independent of the training data. Here, we employ an unsupervised comparison method, Gromov-Wasserstein optimal transport (GWOT), which estimates human-DNN correspondences from the internal distance structure alone and thereby tests fine-grained alignment. We further assess generalization on a curated test set of concepts non-overlapping with the training data. We show that fine-tuning pre-trained DNNs with Relational Knowledge Distillation (RKD), an established relational transfer method, brings DNNs close enough to humans to be aligned at the individual-object level on this test set. We also show that this improvement is driven by a more human-like global structure, as reflected in the ordering of distances among coarse categories, while the local human-DNN nearest-neighbor overlap rate remains largely unchanged. These findings indicate that relational transfer from humans brings the global structure of pre-trained DNNs close enough to the human structure to enable fine-grained human-DNN alignment without supervision.