When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
2026-07-27 • Cryptography and Security
Cryptography and SecurityMachine Learning
AI summaryⓘ
The authors studied different ways to protect large language models from being tricked or misused, called jailbreak defenses. They found that while these defenses help with safety, they also come with trade-offs like making the model refuse to answer harmless questions or slowing it down. Different defense methods have different side effects: rule-based defenses keep performance good, self-checking defenses refuse more harmless inputs, and multi-step defenses take more time. Their work helps understand these trade-offs and guides choosing defenses depending on what matters most, like speed or accuracy.
large language modelsjailbreak defensesmodel performanceover-refusalinference costrule-based defensesself-reflective defensesruntime overheadmodel safetytrade-offs
Authors
Tong Zhang, Zexin Li, Simin Chen, Yun Peng
Abstract
Jailbreak defenses are essential for protecting large language models (LLMs), but they can also introduce secondary costs that weaken model utility. We present a systematic study of these defense trade-offs along three dimensions: performance impact, over-refusal on benign inputs, and inference cost. Rather than treating defenses as a single class, we organize them by operational strategy and examine how different strategies correlate with different side-effect profiles. Across state-of-the-art defense methods, widely used benchmark datasets, and representative open-source LLMs, we find that defenses rarely improve downstream capability, but instead vary in how they trade safety gains against usability and efficiency. In particular, rule-based defenses best preserve task performance, highly conservative self-reflective defenses often increase over-refusal, and multi-round defenses incur the largest runtime overhead. These results provide both a benchmark for evaluating defense side effects and practical guidance for selecting defenses under deployment constraints.