Surprise masking reduces private data leaks in language models
Surprise Masking: Private Data Filtering with Model Likelihoods
Cryptography and Security
Summary
Training language models can accidentally make them memorize private or unwanted information from the data. The authors propose Surprise Masking, a way to use a public model to spot and mask these sensitive parts so the model learns from the rest safely. Their tests show it lowers privacy risks far more than standard methods while keeping good performance on coding tasks. It works best when unwanted parts can be clearly separated from useful data and is especially helpful for code datasets.
What this means in practice
- •For software development teams: Reduce the risk of leaking private data when training or fine-tuning language models on code by selectively masking sensitive information.
- •For data compliance officers: Filter training datasets to remove or mask unwanted content, improving data privacy protections without losing useful examples.
Authors
Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco, Avi Schwarzschild, Om Thakkar
Abstract
Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple form of perplexity filtering that uses a publicly available reference model to identify unwanted spans and suppress learning from them while preserving the rest of each example. On synthetic code benchmarks, we show SM improves the privacy-utility tradeoff over standard masking methods and PII filtering. In fine-tuning experiments, SM reduces approximate extraction of private values from revealed prefixes from 66.3% to 2.3% and the reproduction of stylistic coding quirks from 23.6% to 1.3%, both while preserving performance on the evaluated coding tasks. We also evaluate SM for sanitizing datasets by directly removing unwanted content. Here, SM outperforms DePA, a perplexity-based defense for detecting poisoned dead code, while specialized detectors perform better on broader credential and PII benchmarks. Across eleven settings, we find that SM's downstream success is predictable from the separation of unwanted and useful content in the distribution, and it is particularly useful with code data. Ultimately, SM is an inexpensive, adaptable method to preserve valuable examples while reducing the learning of unwanted information.