Model safety depends on hidden internal signals not just outputs
From Latent Space to Jacobian Space: Measuring, Evading, and Training Against Safety-Content Accessibility
Artificial IntelligenceComputation and Language
Summary
Detecting unsafe content from AI models just by watching their final replies can miss important warning signs hidden inside. The authors studied how internal signals that predict unsafe outputs are connected to what the model is ready to say. They developed new ways to measure how safety-related thoughts inside the model become visible at different layers and how training changes this. Their findings suggest that monitoring and improving AI safety is better done by looking inside the model’s hidden computations rather than only checking its final answers.
What this means in practice
- •For ai safety engineers: Design safety monitors that track internal model signals instead of just outputs to better detect unsafe content.
- •For large language model trainers: Train models with techniques that widen internal safety signal gaps to improve refusal behavior without losing performance.
Authors
Mohammad Mosafer
Abstract
Output-only safety monitoring sees only the end of a model's computation, yet the model computes its answer before emitting it: what it is internally poised to say is safety-critical. Jacobian-space (J-space) readouts, linear maps from hidden states to the output vocabulary via the model's input-output Jacobian, have been proposed as a window on behaviorally accessible internal content, and a first safety protocol (JADR) showed that danger recognition is readable there. What remains unknown is how this accessible content relates to the latent-space safety geometry studied by representation engineering, and how safety training shapes it. We introduce two quantitative bridges: (i) a transport-amplification profile $A_\ell$, measuring how strongly a latent safety direction is carried toward output space by each layer's Jacobian, and (ii) a paired base-vs-tuned protocol that attributes accessibility to training provenance. Across four small model pairs (135M to 1.5B, three lens-fit seeds), tuned checkpoints concentrate recognition in upper-middle layers, and at 0.5B DPO installs refusal while J-space recognition drops to chance: training can widen the accessibility gap exactly where behavioral safety looks best. A deployment audit closes the loop: monitor-aware GCG-style suffixes suppress the prompt-side monitor at zero behavioral cost at every scale; only a learned monitor rung resists its own adaptive re-attack; the training-time defense fails its re-attack at every penalty weight; steering shows the amplification profile is descriptive, not causal; and continuous-prefix optimization fails where discrete search succeeds. Safety monitoring, training, and evaluation must operate on accessibility itself, not on outputs alone.