Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs
2026-08-10 • Machine Learning
Machine LearningArtificial IntelligenceCryptography and Security
AI summaryⓘ
The authors address the problem of large reasoning models being tricked by harmful prompts that make them unsafe. Instead of just blocking known bad prompts, they create a system called AdvSafe that teaches models to understand why and how these tricks work by playing a two-part game: one part generates tricky prompts, and the other explains why they succeed and how to spot them. This approach helps models learn a deeper sense of safety without losing their ability to think well. Their tests show AdvSafe makes models much harder to fool, even with new kinds of attacks, using only a small amount of training data.
Large Reasoning ModelsAdversarial AttacksJailbreak PromptsSafety AlignmentAdversarial SynthesisAdversarial ExtractionRobustnessCognitive DefenseOut-of-distribution PromptsDataset Synthesis
Authors
Hongli Shen, Shaopeng Fu, Qinbo Zhang, Jian Li, Di Wang
Abstract
Large reasoning models (LRMs) achieve remarkable success on complex tasks but remain vulnerable to harmful prompts that induce unsafe outputs. Recent methods align LRMs using direct refusals or safety rationales, yet often focus on prompt patterns rather than intrinsic attack mechanisms. As a result, these pattern-centric alignments struggle to generalize across diverse jailbreaks, compromising adversarial robustness and reasoning utility. We propose AdvSafe, a dual-adversarial framework that enables LRMs to internalize unsafety knowledge by explicitly deconstructing adversarial mechanisms. This moves beyond pattern-dependent traces, fostering robust cognitive defense without compromising reasoning utility. Our pipeline operates via a two-phase adversarial game. First, in adversarial synthesis, an autonomous agent dynamically crafts deceptive jailbreak prompts, adapting its strategies to breach a strong teacher model. Second, in adversarial extraction, the breached teacher executes a cognitive counter-attack. For every successful jailbreak, the teacher unmasks the camouflage, explaining why the attack succeeds and how such prompts can be identified and mitigated. This dual-adversarial process yields a compact reasoning dataset capturing rich, generalizable unsafety knowledge. Student models trained on this dataset implicitly acquire safety alignment through intrinsic threat comprehension. Experiments show that with only 1K synthesized samples, AdvSafe-aligned LRMs achieve significantly stronger jailbreak robustness than existing baselines, with almost no utility degradation. Furthermore, AdvSafe improves robustness against out-of-distribution prompts, demonstrating that learning unsafety knowledge enables a superior robustness-utility trade-off and generalizes beyond seen attack patterns.