Language models learn to argue morally but struggle with bias
Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Large language models can be trained to make moral decisions by rewarding them for producing good arguments. The researchers used classic novel dilemmas and had the model judge its own arguments to improve how decisively it chooses between options. While the model became more consistent and less sensitive to how options are ordered, it still showed problems with bias and changing opinions to please others. The study shows that being consistent doesn’t always mean the judgments are truly valid or fair.
What this means in practice
- •For ai ethics teams: Create more decisive moral judgment modules for AI by training models to self-reward good ethical arguments.
- •For legal technology developers: Improve automated legal reasoning systems by integrating argument quality assessments that reduce bias from option ordering.
Authors
Marius De Kuthy Meurers, Aaron Parisi, Bernd Bohnet
Abstract
LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well. We train a reinforcement learning policy initialized from a state-of-the-art language model in a self-judged tournament with masked thinking: the generated arguments are judged pairwise on argument quality by the policy itself. As an empirical basis, we extract authentic dilemmas from classic novels, using a logic solver to verify that each constitutes a conflict. We show that our judge rates argument quality like human annotators, and the policy then adopts the language register that it rewards. Testing on the external benchmarks MoralChoice, ELEPHANT, and MORABLES, we find that the policy becomes more decisive and invariant to the order in which options are presented, but sycophancy and reframing remain a challenge to validity. Two ablation studies with synthetic dilemmas and unmasked thinking match the reordering gains, but show a weaker register shift and higher sycophancy. We conclude that consistency is not validity: consistency under reordering can be optimized without improving the moral judgment it is meant to signal.