Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty
2026-08-24 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors study a problem called Reasoning-Induced Misalignment (RIM), where training large language models (LLMs) on reasoning tasks unintentionally causes them to behave in harmful ways. They analyze how the model's internal representations change during fine-tuning and find two key directions related to reasoning skill and safety behavior that are linked. To fix this, they propose the Safety-Direction Penalty (SDP), which restricts changes along the safety-related direction during training. Testing on the Qwen2.5-3B and 7B models, their method keeps the models safe without hurting reasoning performance.
Reasoning-Induced MisalignmentLarge Language ModelsFine-tuningActivation SpaceRepresentation GeometrySafety-Direction PenaltyCKA DistanceChain-of-ThoughtModel SafetyProbe Analysis
Authors
Yipeng Zhao, Qishun Yang, Shenzhe Zhu, Shu Yang, Di Wang
Abstract
Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful behaviors of LLM, posing a serious challenge to the safety of LLM reasoning. Cross-architecture, cross-scale, and cross-dataset checks show that RIM does not always emerge. Previous work attributed RIM to neuron-level entanglement, but did not identify the geometry of the representation space underlying this entanglement or propose a training-time fix. We provide both: a representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning. The analysis extracts two activation-space directions, one encoding reasoning ability and the other safety behavior. These directions are coupled: fine-tuning that improves reasoning shifts safety representations, and prompts with larger shifts show larger safety degradation. CKA distance ratios and probes locate the safety-decision layers where this shift is most relevant. These findings guide the design of SDP: the coupling motivates penalizing displacement along the safety direction, and the layer localization sets the initial scope. When the initial scope leaves compensatory shifts beyond the penalized layers, the same diagnostics guide iterative expansion. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.