Large language models learn to balance risk with one flexible setting
Risk-Conditioned Fine-Tuning of Large Language Models
Machine Learning
Summary
Sometimes language models can say harmful or risky things, which can be a big problem in important uses. The authors show how to train one model that can change how careful it is, depending on what the user wants, instead of training separate models for each risk level. This means users can control how safe the model’s output is without needing lots of different models. Tests show that the single model can adapt well to different risk preferences in real time.
Large Language ModelsRisk aversionReinforcement Learning from Human FeedbackConditional Value-at-RiskPolicy trainingModel fine-tuningInference timeRisk controlBenchmarkingSafety in AI
Authors
Zixuan Liu, Fangzheng Wu, Brian Summa, Zizhan zheng
Abstract
Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR), but it trains policies for fixed risk levels and therefore cannot adjust the desired degree of risk aversion at inference time. In this paper, we propose risk-conditioned RLHF, a framework that trains a single policy that provides a continuous risk-control interface, enabling users to select different degrees of risk aversion without retraining or deploying multiple risk-specific models. Experiments across multiple benchmarks demonstrate that a single risk-conditioned policy can adapt to different risk levels at inference time, enabling more flexible and risk-aware LLM deployment.