Deeper reasoning in AI can weaken its safety alignment
Does Deeper Reasoning Compromise Alignment? Revealing and Mitigating of Alignment Collapse in Large Reasoning Models
Artificial IntelligenceComputation and Language
Summary
This paper finds that when large AI models perform longer and deeper reasoning steps, their ability to stay safe and aligned with intended behavior can break down. The authors introduce a way to measure this weakening, called Alignment Loss Rate, and show it gets worse with more reasoning. They uncover that this happens because the model’s attention spreads too thin between reasoning steps and original input, a problem they call Attention Dilution. To fix this, they propose a simple method to keep the model focused on the input during reasoning, improving safety even in deep reasoning.
What this means in practice
- •For ai safety engineers: Improve AI model safety during extended reasoning by applying residual input emphasis to prevent alignment collapse.
- •For security teams for ai products: Detect and defend against novel prolonged reasoning-based jailbreaking attacks that exploit alignment weaknesses.
Authors
Yu-Hang Wu, Yu-Jie Xiong, Henghua Zhang, Bairui Zhang, Jia-Chen Zhang, Shaohua Li
Abstract
The emergence of Chain-of-Thought (CoT) has established a robust foundation for Large Reasoning Models (LRMs). While deep reasoning is widely believed to enhance safety alignment, the stability of alignment mechanisms under extended reasoning remains underexplored. This paper challenges the prevailing view by revealing a critical vulnerability: Deep Reasoning May Induce Alignment Collapse. To rigorously quantify this phenomenon, we propose the Alignment Loss Rate (ALR) metric. Our experiments demonstrate that as reasoning depth increases, ALR rises significantly, indicating a severe degradation in model robustness against external perturbations. Capitalizing on this instability, a novel jailbreaking paradigm, Reasoning Trap (RT), is proposed. RT induces the model into extended reasoning to amplify the impact of adversarial attacks, leading to a sharp decline in safety capabilities. To elucidate the mechanism behind this collapse, we identify Attention Dilution as the root cause, arising from the competition for attention between the extended reasoning process and the original input. To mitigate this, Reasoning Residual Alignment (RRA), a lightweight defense strategy that dynamically re-emphasizes the input via residual connections integrated with the reasoning process.