Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
2026-08-31 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors studied how mixing different types of preference data can affect a language model's tendency to memorize sensitive information. They introduced a method called Privacy-Pressure Preference Mixing (P3M) that adds synthetic privacy-focused data while keeping other helpful and harmless data steady. Their tests showed that this mixing can reduce signs of memorizing sensitive details and make certain privacy attacks less effective, but not consistently across all data types. Accuracy on harmlessness preferences changed depending on the model, while helpfulness accuracy stayed stable. They suggest P3M is useful for exploring privacy and utility trade-offs but is not a formal privacy protection.
Preference optimizationPrivacy-preserving machine learningMemorizationDirect Preference Optimization (DPO)Synthetic dataMembership inference attackLanguage modelsGemma modelsAUROCAUPRC
Authors
Dishu Yang, Jingjing Liu, Jize Li
Abstract
Preference optimization is widely used to align large language models with human preferences, but preference-data composition may also influence privacy-relevant memorization. We examine whether adding synthetic privacy-preference pairs to Direct Preference Optimization (DPO) is associated with lower canary-based memorization signals without modifying the objective or introducing a formal privacy mechanism. We propose Privacy-Pressure Preference Mixing (P3M), a data-composition protocol that varies the amount of privacy-preference data while keeping helpfulness and harmlessness preference data fixed. We evaluate a non-privacy Baseline and privacy-mixing ratios of 0.5, 1.0, and 2.0 using Gemma 3 270M-IT across five random seeds and validate the same four conditions using 4-bit-quantized Gemma 2 2B-IT across three seeds. Overall, under the tested conditions, privacy-preference mixing is associated with lower mean canary suffix log-likelihood proxy values across both model settings and lower aggregate membership-inference attack performance relative to the Baseline in the mixed-source 2B evaluation. Specifically, across the privacy-aware 2B configurations, the mean area under the receiver operating characteristic curve (AUROC) ranges from 0.596 to 0.629, and the mean area under the precision-recall curve (AUPRC) ranges from 0.541 to 0.575, compared with 0.804 and 0.790, respectively, for the Baseline. However, the reduction in membership distinguishability does not hold uniformly across data sources. Moreover, the relationship between the privacy ratio and harmlessness preference accuracy varies by model setting, whereas helpfulness preference accuracy remains broadly stable. These findings suggest that P3M should be viewed as a lightweight empirical protocol for examining privacy-utility-safety trade-offs rather than as a formal privacy guarantee or a defense against extraction attacks.