Subliminal learning explained through kernel-based model dynamics
Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learning
Machine LearningArtificial Intelligence
Summary
Sometimes, a smaller AI model can learn a task by watching a larger model's unrelated outputs, even without seeing the actual task data or labels. The authors explain this surprising effect by analyzing how learning changes happen inside the smaller model using a mathematical tool called kernels. They show how the structure of these kernels ensures the smaller model learns the right task, how the number of 'ghost outputs' limits learning, and why random input data helps the process. Experiments confirm their theory, providing the first clear explanation of how so-called ghost outputs enable this hidden learning.
What this means in practice
- •For machine learning engineers: Design and analyze knowledge transfer methods that train models without explicit task labels using ghost-output supervision.
- •For signal processing teams: Develop probing input signals based on broadband properties to enhance learning across different tasks in models.
Tested on simulated data.
Authors
Zhe Li, Bicheng Ying, Chaosheng Dong, Haibo Yang
Abstract
Subliminal Learning (SL) is a recently identified phenomenon in which a student model acquires downstream task capabilities by matching seemingly unrelated auxiliary outputs from a teacher, despite never observing task labels, task-specific outputs, or the original training data. While recent studies have identified where subliminal signals may reside, the optimization mechanism underlying this phenomenon remains poorly understood. In this work, we provide a mechanistic understanding of SL through the lens of learning dynamics. Specifically, we derive a chained cross-task kernel that explicitly links ghost-output supervision to changes in task predictions through shared backbone representations. Our unified analytical framework provides a rigorous mathematical explanation for three central empirical puzzles in SL: (i) under shared initialization, the transfer operator forms a strictly Positive Semi-Definite (PSD) structure, guaranteeing that ghost-output optimization aligns the student with the teacher's true task objective without explicit label exposure; (ii) the ghost-output dimensionality acts as an explicit rank bottleneck governing the transfer of task-relevant features; and (iii) synthetic, high-entropy inputs function as broadband probes that maximize cross-task kernel overlap, explaining why random noise consistently outperforms structured data for subliminal transfer. Experiments on the canonical ghost-output setting validate all three theoretical predictions, providing the first learning-dynamics-based theoretical explanation of how ghost-output supervision gives rise to subliminal learning.