On-policy distillation students leak membership of teacher training data
Leaky Students: Membership Inference against On-Policy Distillation
Machine LearningCryptography and Security
Summary
When training a small model (student) to mimic a bigger model (teacher) using examples generated by the student, the small model can unintentionally reveal if specific data was used during training. The authors found that using new examples created by the student exposes these membership leaks better than traditional methods. They created a tool called Leaky that detects these leaks by comparing likelihoods between the target and reference models. Tests across different question types show Leaky detects data membership more accurately than existing methods.
What this means in practice
- •For machine learning engineers: Detect whether small models trained via on-policy distillation reveal private data from the original training set of a larger model.
- •For software security teams: Assess privacy risks of deployed distilled models by identifying if training data membership can be inferred from model outputs.
Authors
Zhexi Lu, Mingzhi Zhu, Stacy Patterson, Lei Yu
Abstract
On-policy distillation (OPD) trains a student to match a teacher's next-token distributions on student-generated trajectories. However, privileged information supplied to the teacher for OPD training may contain sensitive data. Whether the student leaks private information about the records supplied to the teacher during distillation remains poorly understood. To the best of our knowledge, we present the first systematic study of membership inference in this setting. We find that fresh student trajectories expose sparse membership signals that fixed reference-answer losses often miss. These signals are mixed with probability changes caused by training on other records. We introduce Leaky, which samples fresh trajectories from the target model and compares its token log-probabilities with the maximum across matched reference models trained without the candidate records. It applies Leaky ReLU to the resulting gaps, preserving positive gaps and downweighting negative gaps as an approximate correction for incidental positive gaps in non-members. Across fifteen targets spanning mathematics, medical question answering, and code generation, Leaky outperforms all evaluated baselines and achieves mean AUROC 0.875, compared with 0.614 for the strongest baseline on each target in the main evaluation. On the same sampled trajectories, the strongest baseline achieves mean AUROC 0.826. These results show that students trained through OPD can expose the membership of records used for teacher supervision, even when fixed reference-answer losses provide little evidence of membership.