On the Failure of Boundary-Seeking Distillation in Bottlenecked Generative Architectures
Machine LearningArtificial Intelligence
Summary
The authors studied how a method called CAKE, which helps teach smaller models by focusing on the decision boundary of bigger models, works for autoencoders. They changed the problem so the autoencoder's output could be treated like a classification task. Their experiments showed that using boundary-focused goals doesn't work well for autoencoders because the outputs are tightly connected and share a limited internal space, causing conflicting learning signals. Instead, they developed a way to respect the autoencoder's learned data structure, which avoided these problems and worked better for data-free training.
Authors
Mohamed Amine Kina
Abstract
Data-free knowledge distillation transfers the knowledge encoded in a teacher model to a student model without access to the original training data. Prior work such as Contrastive Abductive Knowledge Extraction (CAKE) achieves this for classifiers by synthesizing samples near the teacher's decision boundary. In this work, we investigate whether this boundary-seeking principle extends to autoencoder distillation through experiments on the MNIST dataset . To enable a direct comparison, we reformulate continuous reconstruction as a dense, per-feature classification task, allowing the decoder to output categorical logits. We show that boundary-seeking objectives are fundamentally ill-posed in bottlenecked generative architectures. CAKE operates on a single, instance-level objective, but a decoder acts as an array of tightly coupled, feature-level classifiers constrained by a shared low-dimensional bottleneck. Independently sampling contrastive targets for these coupled outputs violates the geometry of the learned latent manifold and produces severe gradient conflicts instead of informative boundary samples. Manifold-aware synthesis bypasses these conflicts entirely and establishes an effective baseline for data-free generative distillation.