Layer patterns reveal how harmful AI prompts develop over processing
Harmfulness Propagation Dynamics: Layer-wise Trajectories of Adversarial Intent in Large Language Models
Computation and Language
Summary
Some prompts given to large AI language models are harmful or dangerous. The authors find that the way harmful intent builds up in the AI’s processing layers follows a clear pattern, rising steadily in strength as the model thinks deeper. They create a lightweight tool called HERALD that watches how harmful signals grow through the layers to detect harmful prompts better than older methods. HERALD can say when harmful ideas appear and how they form inside the AI, helping to understand and catch bad inputs more reliably.
What this means in practice
- •For ai safety teams: Detect adversarial or harmful inputs in language models by monitoring how harmful intent develops across layers for more reliable moderation.
- •For content moderation developers: Build input moderators that classify harmful prompts using compact features derived from layer-wise model activations without heavy computation.
Authors
Noor Islam S. Mohammad, Uluğ Bayazıt
Abstract
We identify \textbf{Harmfulness Propagation Dynamics (HPD)}: for harmful prompts, the projection of the last-token hidden state onto a learned harm direction rises monotonically with transformer depth, whereas benign prompts remain flat or oscillatory. This cross-layer signature reflects harmful intent as a \emph{progressively resolved} semantic property: surface form appears early, while pragmatic intent consolidates later, making the \emph{trajectory shape} more informative than any single-layer snapshot. Moreover, LDA-based harm directions, learned per layer, remain stable across random splits (pairwise cosine similarity $>0.97$), supporting the projection sequence as a reproducible structured signal. Building on HPD, we introduce \textbf{\herald{}} (\textbf{H}armful \textbf{E}ncoding \textbf{R}ecognition via \textbf{A}ctivation \textbf{L}ayer \textbf{D}ynamics). This lightweight input moderator extracts a seven-dimensional feature record, slope, curvature, monotonicity, onset layer, and related statistics from the cross-layer projection sequence and classifies it with a 288-parameter MLP. \herald{} stores one $d$-dimensional direction per layer ($262$\,KB for a 32-layer, $d{=}4096$ model), requires no gradient computation during training, and adds only $2.6{\times}10^{-6}$ prefill FLOPs at inference. Across eight prompt-harmfulness benchmarks and four model families, \herald{} achieves an average F1 of $89.3$ on OLMo2-7B, surpassing all tested guard models on adversarial jailbreak detection ($98.4$ vs.\ $96.9$ F1) and outperforming prior latent-based methods by $2.3$-$4.1$ F1 points on every backbone. Per-instance trajectories provide machine-readable audit records that reveal \emph{when} and \emph{how} harmfulness emerges, offering an interpretability advantage over single-layer approaches.