Assistant preferences influence user turn predictions in chat models
Measuring the Assistant's Harmlessness Preferences on the User Turn
Computation and Language
Summary
When AI chat models learn to act like helpful assistants, their safety preferences—such as avoiding harmful content—also affect what the model predicts the user might say. This happens even though the user is a separate speaker. The authors found this effect grows stronger as the models get bigger and emerges during fine-tuning after the base model is trained. This suggests the model’s sense of safety is built into how it represents both the assistant and the user in conversation.
What this means in practice
- •For language model developers: Improve safety evaluations by measuring how assistant preferences influence predictions on user input turns.
- •For ai safety engineers: Design safer dialogue systems by understanding how safety-focused fine-tuning affects model behavior beyond assistant responses.
Authors
Jord Nguyen
Abstract
Post-training turns a general next-token predictor into a chat model with a persistent assistant persona. If that persona is a character the model plays only on its own turns, its preferences should govern what the assistant says, not what the model predicts other speakers will say. We test this boundary and find that it does not hold: a safety-relevant preference of the assistant---for harmless over harmful tasks---shapes the model's predictions even on the user's turn, where the assistant is not the one speaking. We find that this preference is small or near-zero in pretrained base models, that it emerges through post-training, replicated across open-weight model families, grows with scale, and can be moved by narrow finetuning that never touches user turns. We claim that this is evidence that post-training does not merely install a shallow assistant persona, but instead generalises beyond just the local assistant turn, into the model's representation of the user.