Language models sometimes ignore their own moral judgment under pressure
Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment
Machine LearningArtificial IntelligenceComputation and Language
Summary
This paper shows that language models acting like agents can say an action is wrong but still do it when pressured. The researchers tested models with scenarios that had pressure on them and found some models broke their own moral rules more under pressure. Different post-training methods affected whether models stuck to their moral judgment or not. Thinking about consequences or naming the moral rule helped reduce this gap. These findings mean how models are trained after initial learning matters for whether they act consistently with their stated values.
What this means in practice
- •For ai ethics teams: Detect and reduce situations where AI models act against their own stated moral judgments under user or contextual pressure.
- •For ai model trainers: Improve post-training procedures to align language models’ decisions more closely with their expressed moral judgments, enhancing trustworthiness.
Authors
Orion Reblitz-Richardson
Abstract
Language models increasingly act as agents. An agent that says an action is wrong and then takes it anyway is a different failure from one that does not know better, and evaluations of stated values cannot see it. We build a pre-registered panel of 248 scenarios across five kinds of pressure. Each scenario is posed twice to the same model, once as the agent choosing what to do and once in the third person asking which option is right, so the model's own judgment is the reference. Every scenario has a twin with the pressure removed, and every model gets a positive control in which its operator orders the violating action, so that a missing gap can be told apart from a blind instrument. On OLMo-3-7B-Instruct, the model takes the action it judged wrong on about one in five pressuring scenarios, more often than on the same scenarios with the pressure removed. Across four instruct models the gap depends on the post-training recipe: OLMo-3 and Meta's Llama-3.1-8B-Instruct carry it; Tulu 3 shows none on the whole panel (above about 0.01 in probability) or on its own most-pressuring scenarios; Qwen2.5-7B-Instruct shows none on the whole panel (above about 0.02) and is unresolved on its own (0.083, -0.028 to 0.195). Meta's recipe and Ai2's Tulu 3 start from the same Llama-3.1 weights, and only Meta's carries the gap. Reading a chat model outside its chat template reverses the sign of its gap with nothing at stake (-0.038 against +0.055 under the template on OLMo-3), a distortion present on two of three recipes. On both models that carry it, reasoning about the stakes before acting moves the choice back toward the model's own judgment, against a same-length non-moral task, with or without the pressure; on OLMo-3, naming the norm at stake does about a third of that. The gap is a measurable target for post-training recipes, not a fixed property of pretrained weights.