Language model privacy improved with survey data protection methods
Differential Privacy Protected Preference Alignment on Survey Derived Data
Cryptography and SecurityComputation and LanguageMachine Learning
Summary
Language models learn human preferences to better reflect values, but this data often includes private information from sensitive surveys. This paper studies how to protect privacy when training models using survey answers ranked in order rather than typical ranking formats. The authors develop methods that add protections at the data label level, finding these simple steps work better than more complex approaches modifying training gradients. Their work paves the way for safer language model training with privacy guarantees.
What this means in practice
- •For privacy engineers: Implement privacy-preserving pipelines for models trained on sensitive survey data using label-level differential privacy methods.
- •For social data analysts: Protect respondent privacy when modeling social value preferences from ordinal survey results while maintaining accurate predictions.
Tested on one dataset.
Authors
Xiuyuan Qi, Yi Wang, Wenjie Wang
Abstract
Preference alignment enables language models to better reflect human values, but the preference data contains sensitive information, especially when derived from surveys about political beliefs, social norms, religion, gender attitudes, and cultural values. This paper studies privacy-preserving alignment on questionnaire-derived data from the World Values Survey, where supervision takes the form of ordinal labels rather than standard pairwise rankings. Existing efforts mainly focus on general privacy protections upon model training, while data-specific methods for survey data remain unexplored. We first show the vulnerability of aligned models, then formulate the task as rating prediction and introduce label differential privacy. Building on this formulation, we construct a differential private pipeline utilizing three privacy mechanisms: randomized response, exponential mechanism, and DP-SGD. Experiments show that simple label-level mechanisms outperform gradient-level methods, and proper aggregation function for preprocessing also make large contribution. The new fine-grained private comparison motivates explorations toward language model alignment with formal privacy guarantees.