Fine-tuning language models can reawaken hidden privacy risks
Sleeping Secrets: How Fine-Tuning Reawakens Privacy Risks in Language Models
Cryptography and Security
Summary
Large language models sometimes remember private information from their training. This paper shows that even after training seems to forget this information, a process called fine-tuning can bring back hidden private details. The authors introduce a method called ReGap that recovers these private associations without needing the original private data. This means customizing these models for new tasks can unintentionally expose sensitive information again.
What this means in practice
- •For ml platform teams: Evaluate privacy risks before deploying fine-tuned language models to prevent unintended data exposure.
- •For security operation centers: Detect and mitigate privacy leakage from customized language models in production environments.
Authors
Jianhong Li, Jiahao Chen, Yuwen Pu, Chunyi Zhou, Oubo Ma, Zhou Feng, Hangtao Zhang, Jichao Bi, Chunqiang Hu
Abstract
Beyond adapting Large Language Models (LLMs) to specialized applications, fine-tuning has recently been shown to recover private information that is no longer accessible through direct queries. Previous fine-tuning recovery attacks, however, require genuine private supervision drawn from the same distribution, i.e., the previous training dataset. We argue that such recovery remains possible without such impractical knowledge. We show that LLM-generated candidates can provide sufficient supervision to recover previously learned private associations. Based on this, we propose ReGap, a data-free attack that recovers private associations using task structure, filters them by answer-token likelihood, and updates the target model via low-rank adaptation. Specifically, ReGap requires neither target answers nor auxiliary genuine private supervision. Across six GPT-2, OPT, and Qwen3 models, ReGap improves target-association recovery by 6-21 percentage points over the post-training target model. Recovery remains substantial even when the adaptation identities are disjoint from all memorized and evaluation identities, with no exact target answers appearing in the generated or selected supervision. Moreover, the same trained adapters increase recovery from 42\% to 63\% on a previously exposed checkpoint, but produce no gain on a matched checkpoint that never encountered the targets. This contrast shows that adaptation alone is insufficient to explain the observed recovery and that prior target exposure strongly affects post-adaptation recoverability. Our findings highlight that routine model customization can reawaken latent privacy risks, warranting urgent attention from the academic and industrial communities.