Crowdsourced fine-tuning can increase private data leaks in language models
The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoning
Cryptography and SecurityMachine Learning
Summary
Fine-tuning large language models often uses conversations from many users to teach the model new tasks. The authors found that a bad actor can sneak in a small number of poisoned examples during this crowd collection. These poisoned examples make the model reveal much more of other users’ private data than it normally would. They also show common ways to find and block poisoned data don’t work well. This means sharing data for training can unintentionally put others’ secrets at risk without easy detection.
What this means in practice
- •For machine learning engineers: Detect and mitigate privacy risks when crowdsourcing data for fine-tuning large language models.
- •For security teams: Design monitoring systems to identify subtle data poisoning attempts that increase leakage of private instructions in deployed models.
Authors
Sae Furukawa, Alina Oprea
Abstract
Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks. Crowdsourcing user conversations is an established approach to collecting SFT data at scale while reducing the need for costly manual annotation. However, it also allows untrusted users to contribute data to the fine-tuning pipeline. We investigate an underexplored privacy risk arising from this setting: can a malicious user poison a small fraction of the crowdsourced data to amplify extraction of previously unseen instructions contributed by other users? We show that this is possible using only black-box, output-only access to the deployed model. Experiments across four models and two datasets demonstrate substantial increases in training-data extraction: with only 50 poisoned examples, near-verbatim extraction reaches $3.71\times$ the rate without poisoning for Qwen2.5-14B on OpenMathInstruct and $3.08\times$ for Llama-3.1-8B on AceReason. Data filtering also proves largely ineffective in detecting poisoned samples: even the best-performing method achieves only 0.378 in F-1 score, leaving the majority of poisoned samples undetected. These findings demonstrate that seemingly benign crowdsourced contributions can amplify leakage of other records while remaining difficult to identify through data filtering.