AI summaryⓘ
The authors address a problem where existing methods for transferring knowledge from a big neural network (teacher) to a smaller one (student) without original training data rely on information tied to specific model types, which doesn't work well for newer models like Vision Transformers. They propose UniDFKD, a new approach that uses general-purpose semantic guidance based on language and spatial information instead of model-specific stats. This method improves how data is generated and how knowledge is transferred by focusing on what to create, where to place important features, and how to align the teacher and student models spatially. Their experiments show UniDFKD works much better than previous methods across various model types.
Data-Free Knowledge DistillationKnowledge DistillationVision TransformersBatch Normalization statisticsSemantic PriorsSpatial AttributionCategorical Semantic ConditioningStudent-Teacher ModelsGenerative Data Synthesis
Authors
Xuewan He, Tong Chu, Zihan Cheng, Yuchen Su, Qianxin Xia, Guoming Lu, Jielei Wang, Wen Li
Abstract
Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose \emph{UniDFKD}, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. \emph{UniDFKD} governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines \emph{what} to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates \emph{where} evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls \emph{how} knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20\% in both homogeneous and heterogeneous settings.