Attention detects harm and MLP writes refusals in large language models
Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
Artificial Intelligence
Summary
Large language models (LLMs) sometimes refuse safe requests or fail to block harmful ones, which is a challenge for making them safer. This paper finds that the part of the model called 'attention' is better at spotting harmful content, while another part called 'MLP' is more responsible for actually writing refusals. The researchers also show that combining signals from both parts helps better detect and handle harmful or over-rejected requests. They created a system called Sentinel that uses these insights to reduce mistakes compared to using only one signal.
What this means in practice
- •For llm developers: Improve LLM safety by using combined attention and MLP signals to reduce harmful outputs and unnecessary refusals in deployed models.
- •For ai safety engineers: Diagnose and mitigate jailbreak vulnerabilities in LLMs by tracing harmful content recognition through attention and refusal generation through MLP modules.
Authors
Zixuan Weng, Haoqi He, Weifei Jin, Yue Huang, Yang Bai
Abstract
Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires distinguishing harmfulness recognition from the computation that writes refusal. We reframe this as a pathway question and make it measurable on residual-stream updates in decoder-only Transformers, separating the harmfulness information a module update exposes from the change it produces in readable refusal evidence. Across 18 Base-Instruct pairs spanning Qwen, Llama, and Gemma, harmfulness is decoded more readily from attention updates, particularly in early layers, whereas MLP updates contribute more strongly to harm-selective refusal writing in aligned models. Post-training strengthens MLP refusal writing in all 18 pairs without systematically improving harmfulness decodability, while positive MLP writing becomes more concentrated in deeper layers after post-training. Across six reasoning-enabled models, harmfulness remains decodable during reasoning while strong refusal writing concentrates at answer onset, a pattern that persists with reasoning disabled. Causal interventions further support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. Building on these findings, we introduce Sentinel, which combines an attention-derived perception score P with an MLP-derived decision score D. Their joint profiles characterize jailbreaks and over-refusals, and conditional routing with both signals achieves a lower aggregate error rate than single-signal routing and evaluated fixed reminders across eight models.