Pairwise likelihood ratios improve membership inference attacks on machine learning models
Membership Inference via Pairwise Likelihood Ratios
Machine Learning
Summary
Membership inference attacks try to figure out if a machine learning model has seen a particular data point during its training. The paper proposes a new method that compares how likely a data point’s outputs are relative to other points that were not used in training. By combining all these comparisons in a clever way, the method makes stronger and more trustworthy guesses about membership. The authors show that their approach works better than previous methods, especially when it’s important to avoid false alarms.
What this means in practice
- •For machine learning privacy teams: Use pairwise likelihood ratio methods to audit models for data privacy leaks, improving detection of whether training data was exposed.
- •For security analysts: Enhance privacy risk assessments of deployed AI models by applying calibrated hypothesis tests that better balance true positives and false positives.
Authors
Shengjie Niu, Zebin Yun, Yeheng Ge, Jian Huang
Abstract
Membership inference attacks (MIAs) are the standard tool for auditing the privacy risks of machine learning models. Given a query point, an MIA aims to determine whether that point was used to train the target model. In practice, such inference must rely on the statistical signals exposed by the model's outputs, such as confidence scores, logits, and intermediate feature representations. However, existing methods often fail to efficiently summarize and combine these statistical signals. To address this limitation, we propose Pairwise Likelihood MIA (PL-MIA), a unified method that combines a Gaussian likelihood-ratio (GLR) statistic with population calibration and the Cauchy combination test. We characterize theoretically how the GLR retains variance-contraction signals and establish conditions under which population calibration and Cauchy combination improve attack power. We obtain $p$-values from pairwise comparisons between the query point and reference points not used for training, and aggregate these continuous signals using the Cauchy combination test. This preserves the evidence strength that is discarded when each pairwise comparison is reduced to a binary vote. Extensive experiments demonstrate that PL-MIA outperforms strong baselines, improving the true positive rate (TPR) by over 25\% in the critical low-false-positive regime, corroborating our theoretical findings. These results demonstrate how statistical principles can turn noisy model outputs into more powerful, calibrated, and reproducible evidence for membership privacy auditing.