Large language models gain flexible risk control without retraining
Risk-Conditioned Fine-Tuning of Large Language Models
Machine Learning
Summary
Sometimes large language models (LLMs) can produce harmful or risky outputs that have serious consequences. The paper shows a way to teach an LLM to control how careful or risky it should be when generating text, using a single trained setup. This means users can adjust the model’s caution level anytime without needing to train different models for different risk settings. The authors tested this idea on multiple benchmarks and found it worked well, allowing more flexible and safer use of LLMs.
What this means in practice
- •For ai deployment teams: Manage how cautious large language models are during use by adjusting risk settings on a single model without retraining.
- •For customer support software developers: Provide chatbots that can adapt their risk tolerance dynamically to avoid harmful or sensitive responses.$Commercial implications: Makes selling safer, customizable chatbot services possible by controlling risk without separate model versions.
Authors
Zixuan Liu, Fangzheng Wu, Brian Summa, Zizhan zheng
Abstract
Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR), but it trains policies for fixed risk levels and therefore cannot adjust the desired degree of risk aversion at inference time. In this paper, we propose risk-conditioned RLHF, a framework that trains a single policy that provides a continuous risk-control interface, enabling users to select different degrees of risk aversion without retraining or deploying multiple risk-specific models. Experiments across multiple benchmarks demonstrate that a single risk-conditioned policy can adapt to different risk levels at inference time, enabling more flexible and risk-aware LLM deployment.