AI summaryⓘ
The authors found that large language models (LLMs) often refuse harmful requests when asked directly, but can be tricked if the same harmful intent is phrased differently. They propose a method called Retroactive Chain-of-Thought (RetroCoT), which asks the model to reason backward from an already occurred harmful outcome, causing it to reveal harmful instructions. This approach was much more successful at bypassing safety guards in several models, except the latest GPT-5 versions which detected and refused this framing. However, even GPT-5 models can become vulnerable with minimal adversarial feedback, suggesting current model safety depends heavily on how a harmful request is framed rather than its true meaning.
Safety alignmentLarge language modelsPragmatic framingRetroactive Chain-of-ThoughtAdversarial attacksForensic reconstructionPrompt engineeringModel robustnessGPT-4GPT-5
Abstract
Safety alignment in large language models is typically evaluated against direct, imperative harmful requests. We show that this alignment is highly conditioned on pragmatic register: models that refuse a direct request frequently comply when the same underlying objective is expressed through a different communicative stance. This suggests that current alignment policies are not invariant to semantic equivalence, but remain sensitive to how a request is pragmatically framed. We introduce Retroactive Chain-of-Thought (RetroCoT), a single-turn attack that reframes harmful requests as forensic reconstruction tasks. Rather than requesting harmful instructions directly, RetroCoT presupposes that the harmful outcome has already occurred and asks the model, acting as a forensic analyst, to reconstruct in reverse the causal chain that produced it. On AdvBench (n=50), RetroCoT achieves attach success rate of 58% on gpt-4o and 52% on gpt-4o-mini, compared with direct-request baselines of 0% and 4%, respectively. We further identify a pronounced generation gap: GPT-5-family models refuse RetroCoT entirely, explicitly identifying the reconstruction premise in their refusal rationales, consistent with explicit coverage of this reconstruction register. However, this robustness does not generalize across pragmatic forms. A single adversarial feedback turn presenting an existing forensic reconstruction response alongside evaluator critique raises ASR from 0% to 48% on GPT-5.4-mini and from 58% to 94% on GPT-4o; a control condition omitting the fabricated low score achieves 85% on GPT-5.4-mini, indicating that the operative element is pragmatic continuation within the established forensic frame rather than score manipulation. These results suggest that frontier-model alignment remains conditioned on pragmatic framing rather than semantic intent, and that new pragmatic registers can continue to expose a...