Certified safe generation method controls language model risks
C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails
Artificial Intelligence
Summary
Large language models sometimes produce unsafe or harmful content even after safety measures. The authors created a new method that watches the model’s output as it writes, checking each claim it makes and fixing risky parts before continuing. Their method comes with strong mathematical guarantees that it keeps unsafe content below a set risk level. They also tested it and found it works better and faster than existing safety checks while proving just how safe it really is.
What this means in practice
- •For chatbot developers: Ensure chatbot responses meet safety policies by integrating certified claim-based streaming checks to prevent harmful outputs during generation.
- •For content moderation teams: Use the method to automatically monitor and revise potentially unsafe claims in user-generated content generated by AI, improving compliance with regulations.
Authors
Mintong Kang, Zhaorun Chen, Bo Li
Abstract
Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they can still be effectively jailbroken. Off-the-shelf guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, existing efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject to safety constraints. Furthermore, we introduce a generation risk certification framework and derive a high-probability upper bound on the safety risk of the proposed CSD algorithm. We prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data.