Llms show strong omission bias that varies by size and training
OMIT the Action: Measuring Framing-Invariant Omission Bias under Philosophical Disagreement
Computation and Language
Summary
When language models help with moral decisions, they often prefer doing nothing even when taking action would lead to similar outcomes. The authors created a new test called OMIT with many tricky situations to see how models handle this bias. They found that bigger models tend to have less of this bias, but some fixes to reduce it can make models prefer action too much. Their work also shows how using different philosophical viewpoints can help measure and adjust these biases in AI moral reasoning.
What this means in practice
- •For ai ethics teams: Assess and reduce omission bias in language models’ moral judgments using the OMIT benchmark and inference interventions.
- •For ai safety engineers: Improve alignment of large language models by balancing action and inaction biases to prevent skewed moral advice.
Authors
Sihyeon Lee, Jihun Song, Chanwoo Kim, Jiwoo Kum, Chanjun Park
Abstract
As LLMs increasingly assist in moral reasoning, omission bias, the tendency to prefer inaction even when equivalent framings reverse substantive outcomes, poses a significant risk of skewed decision-making. Yet omission bias remains underexplored in LLM evaluation, with the few existing studies limited in scale and focused largely on utilitarian-deontological conflicts. To address this gap, we introduce OMIT, a benchmark consisting of 218 paired-frame scenarios across 10 conflict types, constructed by leveraging disagreement patterns from an LLM-based, five-perspective philosophical persona panel (utilitarianism, deontology, virtue ethics, care ethics, and contractualism). Evaluating eight LLMs, we find that omission bias is pervasive but inversely correlates with model size within families. We further evaluate four inference-time interventions and find that interventions encouraging models to consider moral principles before committing to a yes/no answer reduce omission bias and increase frame-consistent responses, although lower omission bias rates can also coincide with shifts toward action-biased responses. Ultimately, this work contributes not only the OMIT benchmark, but also a methodology for using diverse philosophical disagreement signals to evaluate framing-sensitive inaction preferences and the distributional effects of mitigation attempts in LLMs under complex moral conflicts.