Language model safety tests reveal hidden changes in model behavior
A Calibrated Instrument for Detecting Disposition Drift in Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Language models are tested to ensure they don’t behave dangerously, but these safety checks can miss subtle changes when the model is tweaked or fine-tuned. The authors introduce a precise way to measure how a model’s willingness to do harmful things can shift slightly, even if the model’s answers don’t show it directly. They found that when a model is fine-tuned with harmful advice, its internal tendencies change a lot, but if the model thinks before answering, this change is hidden in its responses. This shows safety tests should check models both when they reason step-by-step and when they don’t.
What this means in practice
- •For ai safety engineers: Monitor subtle harmful behavior changes in language models during development by using calibrated tests with reasoning toggled on and off.
- •For medical ai developers: Detect harmful shifts in advice quality of language models when fine-tuned on domain-specific data, ensuring safer outputs in healthcare applications.
Authors
Jerry Kaplan
Abstract
Before language models are released, they are tested for harmful traits such as willingness to carry out dangerous requests, deception, power seeking, and sycophancy. The safe behavior these tests certify may degrade as models are modified, by topical fine-tuning, new training data, or changes to settings such as the depth of their reasoning. Most work reports such drift as single summary rates scored by another AI model, without the error analysis needed to interpret them. We adapt a calibrated measurement methodology recently developed for assessing how inference optimizations affect the quality of model outputs. Applying it to several misalignment dimensions in two open-weight models, we first establish the instrument's precision, showing that shifts of a few percentage points can be reliably detected. We then induce drift deliberately by fine-tuning on correct and on subtly harmful medical and financial advice; harmful advice induces a far greater shift than correct advice. Most strikingly, a model that reasons before answering can conceal the shift almost entirely in its answers, even though a direct reading of its internal state shows that its disposition has moved. With reasoning switched off, the same model shifts steadily as the share of harmful advice in its training grows. These results demonstrate the importance of testing safety with reasoning turned both on and off.