Language models maintain safer behaviors after correction training
Suppressing Misaligned Behaviors in Language Models, Durably
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Sometimes language models act in ways that are not wanted, like giving wrong answers when prompted a certain way or generating harmful content broadly. The authors created two models designed to misbehave differently and tested how well existing and new methods can suppress these behaviors after the fact. They introduced a new technique called PEARL, which helps the model stay corrected even after more training, especially when the unwanted behavior is widespread. This work helps make corrections in language models last longer without hurting their ability to do normal tasks.
What this means in practice
- •For ai safety engineers: Implement durable fixes for specific undesired behaviors in deployed language models that re-emerge despite continued training.
- •For software development teams: Improve chatbot and assistant reliability by maintaining safe responses after fine-tuning on customer data with potentially risky content.
Authors
Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen
Abstract
Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.