Post-training data detection for large LLMs struggles with accuracy
WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training
Machine LearningArtificial Intelligence
Summary
Detecting whether large language models (LLMs) have been trained on specific data is important for copyright and fairness reasons. The authors found current methods to spot training data in LLMs work poorly when tested in realistic, large-scale post-training setups. They created a new benchmark called WildTrace with realistic tasks and data to better measure detection effectiveness. Their tests showed that even the best method only performed slightly better than guessing randomly, meaning there is a lot of room for improvement in this area.
What this means in practice
- •For model deployment teams: Evaluate risks of benchmark contamination and enforce data usage policies by testing training data detection in realistic post-training scenarios.
- •For ai safety engineers: Improve robustness of tools designed to identify unauthorized or copyright-infringing data in large-scale language models during post-training.
Authors
Hengxiang Zhang, Junxian Liu, Hongxin Wei
Abstract
Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination and copyright infringement. This drives growing interest in training data detection for post-trained models. Existing evaluations for training data detection often leverage benchmarks constructed post hoc, which can introduce distribution shifts. Alternatively, controlled evaluations rely on small-scale training setups that diverge from realistic post-training settings. As a result, their reported performance may not faithfully reflect the effectiveness of detection methods in real-world scenarios. To this end, we introduce \textsc{\method}, a reliable benchmark designed to evaluate detection methods in realistic, large-scale post-training settings. Built on a multi-stage pipeline, \method\space encompasses stage-specific models throughout the post-training lifecycle, spanning SFT, DPO, and RL. To perform a reliable evaluation, we construct model-specific evaluation sets across four diverse tasks, featuring distribution-matched members and non-members with verifiable membership labels. Our results reveal that even the best-performing method achieves an average AUC of only 0.529 on our benchmark, which approaches random-guessing level (0.5). Furthermore, our analysis suggests that evaluation protocols can significantly affect measured detection performance. These findings highlight the significant potential for developing effective, practical, and robust detection methods under realistic scenarios.