Summary
Membership inference tries to tell if a piece of text was used to train a large language model without looking at the training data itself. The authors created a new, carefully controlled test called OLMo-Detect to better evaluate how well these techniques work at different stages of model training and with different types of data. They found that membership detection is harder than expected, performs best in the middle of training with certain data types, improves with bigger models but plateaus, and struggles under changes in data distributions. These patterns hold true for different versions of the OLMo models and other models too.
What this means in practice
- •For machine learning engineers: Evaluate membership inference risks during different training phases using the OLMo-Detect benchmark to improve data privacy assessments.
- •For security analysts: Test robustness of membership inference attacks against distribution shifts using OLMo-Detect (Shifted) to better protect language model datasets.
Authors
Tao Shi, Chaoyi Xiang, Qiongkai Xu, Jey Han Lau
Abstract
Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer from three limitations: limited coverage of training stages, insufficient distributional alignment between members and non-members, and lack of rigorous filtering of non-members against the training corpus. To address these limitations, we propose OLMo-Detect, a multi-stage, confounder-controlled benchmark built upon the fully open OLMo 2 pipeline. OLMo-Detect spans pre-training, mid-training, and post-training, explicitly aligns members and non-members on three key axes, and rigorously filters non-members via infini-gram. To assess robustness to distribution shifts, we further introduce OLMo-Detect (Shifted), a variant where members are misaligned with non-members. We evaluate 15 unsupervised and 3 supervised membership inference attacks (MIAs) across the OLMo 2 family, finding that: (i) overall performance is limited: the best unsupervised and supervised MIAs both reach an AUC of only 0.68, and supervised MIAs degrade under cross-domain evaluation; (ii) MIA performance peaks at mid-training and is lower at pre-training and post-training, a pattern driven by data type rather than a stage effect: curated math data is far more detectable than other types; (iii) overall scores improve from 1B to 13B but plateau at 32B; and (iv) no unsupervised MIA is robust to distribution shifts, with AUCs shifting by up to 0.42. Finally, we find that our findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.