Language models learn from rewards while keeping prompts private
Reward-Driven Learning under Prompt-Level Differential Privacy
Machine LearningCryptography and Security
Summary
Training language models with feedback (rewards) can reveal sensitive information about the training data. The authors developed a new way to protect each training prompt’s privacy by adding controlled noise to the learning process. This method lets models learn from rewards while ensuring that individual prompts can’t be identified from the final model. Their approach improved math and reasoning tasks, even under strict privacy limits.
What this means in practice
- •For machine learning engineers: Train language models on sensitive data with verifiable privacy guarantees for each training prompt under reward-driven learning.
- •For ai model deployment teams: Ensure compliance by releasing language model weights that protect individual prompt data privacy during fine-tuning with rewards.
- •For enterprise ai developers: Build private AI services that incorporate verifier feedback without exposing confidential training prompts, enhancing data privacy.$Commercial implications: Enables private fine-tuned language model products that securely integrate proprietary data feedback, meeting privacy regulations.
Authors
Jiachen Zhao, Antonia Januszewicz, Taeho Jung
Abstract
Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the trained model can reveal which problems it saw. We study RLVR under prompt-level differential privacy: the released weights must be (ε,δ)-differentially private with respect to the presence of any one training problem. Taking the group of responses to one prompt as the privacy record, our method aggregates their gradients, clips the prompt's contribution once, adds Gaussian noise, and composes the privacy loss across updates, so the budget depends on neither the number of responses per prompt nor the clipping norm; to our knowledge this is the first differential privacy guarantee for RLVR training. We train Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of ε=8 and compare, on the same prompts and at the same budget, a control that removes only the reward signal and two private supervised fine-tuning recipes. The reward signal improves accuracy over the control by 2.65 points on MATH and 3.24 on GSM8K, in every seed; the improvement survives a format-robust scorer, at 1.3 points on MATH, and is not explained by response length. At the same budget the private model outperforms both supervised recipes on MATH and GSM8K by 2.3 to 3.8 points, retains 85--90% of the gain of non-private GRPO on these tasks, and on MATH the noise of an eightfold tighter budget costs at most 1.2 points. The reward effect also carries to CommonsenseQA, an exploratory non-mathematical task. Verifier feedback thus remains a usable learning signal under prompt-level privacy.