Large language models reduce unnecessary refusals while keeping safety intact
Revisiting Over-refusal in Safety-aligned Large Language Models: A Representation and Optimization Perspective
Artificial Intelligence
Summary
Large language models sometimes refuse to answer harmless questions because they look similar to harmful ones. The authors studied why this happens by looking at how the model represents and learns from these questions. They found that as the model processes information deeper, it struggles to clearly separate harmless from harmful questions, leading to too many refusals. To fix this, they created a new method called GOAL that helps the model better recognize harmless questions and reduces unnecessary refusals without sacrificing safety.
What this means in practice
- •For ai platform developers: Improve content moderation by reducing false refusals of safe user queries without compromising safety.
- •For chatbot developers: Enhance user experience by lowering unnecessary refusals for harmless questions in conversational AI systems.
Authors
Chao Chen, Pengyang Shao, Wei Qin, Shijie Hao, Fei Shen, Richang Hong
Abstract
Over-refusal is a critical side effect of safety alignment in large language models (LLMs), causing harmless but superficially risky queries to be mistakenly rejected. Despite the effectiveness of existing mitigation methods, why safety-aligned LLMs may still reject harmless queries under correct safety supervision remains insufficiently understood. To address this gap, we analyze over-refusal through the lens of representation and optimization. We make two observations: 1) the representations of harmful and over-refusal samples are visually distinguishable at certain intermediate layers but increasingly overlap in deeper layers; and 2) harmful-sample gradients exhibit high intra-class similarity and concentrate in a low-rank subspace. Moreover, they are more strongly aligned with refusal-target gradients than with correct helpful-target gradients on over-refusal prompts. This imbalanced gradient coupling can bias over-refusal samples toward refusal, while the reduced apparent representational separation in deeper layers may make it more difficult for the model to maintain a clear boundary between harmful and over-refusal samples. To this end, we propose GOAL (Gradient Orthogonalization with contrAstive Learning), a safety-alignment framework for mitigating over-refusal. Specifically, GOAL consists of two components: 1) to prevent imbalanced gradient coupling from biasing the optimization of over-refusal samples toward refusal, we orthogonalize over-refusal gradients against the low-rank refusal subspace extracted from harmful gradients; and 2) to explicitly enhance the separation between harmful and over-refusal representations, we introduce a contrastive learning objective. Extensive experiments across multiple models and benchmarks demonstrate that GOAL effectively mitigates over-refusal while preserving safety performance and general capabilities.