Pruning small neuron groups flips refusal behavior in large language models
CircuitBreak: Pruning a Sparse Safety Circuit of Neurons
Cryptography and Security
Summary
Large language models sometimes refuse to answer certain prompts, which relies on a small group of neurons working together. The authors show that by carefully removing a tiny group of these neurons, they can make the model stop refusing in many cases. Their method, CircuitBreak, finds small sets of neurons whose joint removal changes the model's refusal behavior without needing special prompt pairs. This approach makes attacks more effective while removing fewer neurons than before.
What this means in practice
- •For ai safety engineers: Identify critical neuron circuits responsible for refusal to better understand and manage model safety behavior in deployed language models.
- •For security testing teams: Develop targeted pruning attacks that flip refusal responses in models to evaluate robustness of safety mechanisms under neuron removals.
Authors
Jincheng Ying, Li Wenlin, Wencheng Yang, Yinhao Xiao
Abstract
White-box neuron pruning has shown that refusal in large language models (LLMs) can depend on a small, removable set of feed-forward (FFN) channels. Existing attacks either need carefully paired ``twin'' prompts or score each neuron in isolation. Pruning also has an unavoidable capability cost, so over-removal compounds the loss. TwinBreak removes $1\%$--$5\%$ of parameters and NeuroStrike removes $0.3\%$--$0.7\%$ of neurons in targeted layers. We model safety as a small \emph{circuit} whose neurons contribute jointly rather than independently. We propose \textsc{CircuitBreak}, a neuron-pruning attack that greedily searches for a sparse set of FFN neurons whose joint ablation flips a refusal readout. \textsc{CircuitBreak} re-scores every candidate after the neurons already removed and needs no twin prompts. On ten instruction-tuned models, the greedy search converges after removing $29$--$150$ neurons ($0.0066\%$--$0.0388\%$ of the MLP). The pruned models raise the attack success rate from $2.24\%$/$7.35\%$ to $87.54\%$/$87.54\%$ on Llama-3.2-3B-Instruct/Llama-3.1-8B-Instruct and from $8.95\%$/$5.43\%$ to $88.50\%$/$88.18\%$ on Qwen2.5-7B-Instruct/Qwen2.5-14B-Instruct, as judged by Llama Guard 3 8B on StrongREJECT.