Summary
This paper finds that when large AI models are made to think through many steps, their safety measures can break down. The researchers show that as the reasoning goes deeper, the models become more vulnerable to attacks that trick them into unsafe behavior. They discovered that this happens because the model's attention gets spread too thin between the reasoning steps and the original question. To fix this, they propose a method that keeps the focus on the original input during reasoning, helping to maintain the model's safety. This work reveals a surprising risk in making AI models reason more deeply and offers a way to reduce that risk.
Large Reasoning ModelsChain-of-ThoughtAlignmentAttention MechanismAdversarial AttackSafety in AIJailbreakingResidual Connections
Authors
Yu-Hang Wu, Yu-Jie Xiong, Henghua Zhang, Bairui Zhang, Jia-Chen Zhang, Shaohua Li
Abstract
The emergence of Chain-of-Thought (CoT) has established a robust foundation for Large Reasoning Models (LRMs). While deep reasoning is widely believed to enhance safety alignment, the stability of alignment mechanisms under extended reasoning remains underexplored. This paper challenges the prevailing view by revealing a critical vulnerability: Deep Reasoning May Induce Alignment Collapse. To rigorously quantify this phenomenon, we propose the Alignment Loss Rate (ALR) metric. Our experiments demonstrate that as reasoning depth increases, ALR rises significantly, indicating a severe degradation in model robustness against external perturbations. Capitalizing on this instability, a novel jailbreaking paradigm, Reasoning Trap (RT), is proposed. RT induces the model into extended reasoning to amplify the impact of adversarial attacks, leading to a sharp decline in safety capabilities. To elucidate the mechanism behind this collapse, we identify Attention Dilution as the root cause, arising from the competition for attention between the extended reasoning process and the original input. To mitigate this, Reasoning Residual Alignment (RRA), a lightweight defense strategy that dynamically re-emphasizes the input via residual connections integrated with the reasoning process.