Suan improves large language model safety while keeping responses useful
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Machine LearningArtificial Intelligence
Summary
Large language models sometimes give unsafe or unhelpful answers. The authors found that current open models often refuse to answer too much or lose quality after safety training. They created Suan, a new method that changes the model's training process to be clearer and more stable. Tests show Suan helps models be safer without making their answers worse.
Large Language ModelsSafety alignmentPreference optimizationGradient optimizationResponse qualityModel trainingOver-refusalOpen-weight models
Authors
Oleksandr Cherednichenko, Roman Klypa
Abstract
Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.