Activation monitors benefit slightly from original model computations
How Much Do Activation Monitors Benefit from Original Model Computations?
Machine LearningArtificial Intelligence
Summary
It can be helpful to watch what happens inside an AI model to catch bad behavior early, rather than just looking at its input and output. The authors studied how much advantage these monitors get by seeing the AI’s original inner workings compared to just using modified or simpler versions of that information. They found that while much of the monitor’s effectiveness can be maintained without the original internal details, having access to the true original computations gives a small but consistent improvement. This means monitors partly rely on those original inner signals to do their best job catching changes in behavior.
What this means in practice
- •For ai system developers: Improve AI safety tools by incorporating native activation data to enhance detection of subtle behavior changes.
- •For security analysts: Use activation monitors with or without original computations to spot and respond to undesirable AI output variations.
Authors
James Oldfield, Shawn Im, Jay J Culligan, Philip Torr, Sharon Li, Adel Bibi
Abstract
One powerful way to safeguard AI systems is to monitor their internal computations--where signs of undesirable behavior might be more easily detectable than from models' inputs and outputs alone. However, recent studies begin to question the premise that activations provide a unique window into model behavior. In this paper, we study how much of activation monitors' success depends on access to the original behavior-generating computations. Within a fixed choice of target model and monitoring architecture, we compare how well monitors trained and evaluated on trajectories that differ in computational context can predict the behavior of the original execution, after the transcript is changed in a controlled manner (e.g., its exact text, or chat formatting). We first find that much performance can be recovered without access to the original behavior-generating trace (in up to 4 models across 2 counterfactual monitoring tasks), but a mild yet consistent advantage to native-trained monitors remains, outperforming strongest controls by up to 0.014 AUROC. Interestingly, we find monitors trained with native access have notable uplift on original examples whose behavioral label changes under paraphrasing. We further find this a useful lens to expose limitations of baselines that use simple forms of privileged information (i.e., answer logit margins). While our results show the advantage of native monitor access is modest in aggregate, they highlight the importance of evaluations that identify if and when original model computations provide additional predictive value.