Jev style system one models do not reduce selection bias in decisions
Understanding Jev-Style System One Models: Does Structure Reduce Selection Bias?
Computation and Language
Summary
Selection bias happens when language models prefer some answer options just because of the order they see them. This paper compares a special type of model called Jev-style models against regular language models to see if their decision setup reduces this bias. The researchers tested many models by shuffling answer options to check if correct answers stay stable. They found that while accuracy stayed about the same, the sets of questions where models failed grew a lot when answers were shuffled. This means that Jev-style models do not reliably guard against selection bias any more than usual language models do.
What this means in practice
- •For ai developers: Design evaluation methods that test model answer stability across varied option orders to better identify selection bias in language models.
- •For qa system engineers: Improve multi-choice question answering by recognizing that consistent accuracy alone doesn't guarantee resistance to biases from answer ordering.
Authors
Mingyang Song, Mao Zheng, Zhiyuan Ma, Xinpeng Zhou
Abstract
Selection bias in large language models (LLMs) can persist despite instructions to ignore presentation order or prompts that elicit reasoning. Recent efforts to reproduce Jev-style System One behavior have produced a growing family of models that predict probabilities directly over supplied candidates. Yet systematic comparisons of selection bias across Jev versions, Jev reproductions, and generative LLMs remain limited in multi-option tasks. This leaves a central question: does Jev-style decision structure reduce selection bias? We address this question with a simple test that shuffles options while keeping question and candidate content fixed. Across two Jev versions, twelve Jev reproductions, and six generative LLMs, we trace individual answers through eight presentations and compare them with fixed-order repetition. On our primary collection of 1,700 questions from two benchmarks, Jev's accuracy varies by less than two percentage points, yet its cumulative failure set expands by 27--29\% relative to the original order, with first failures continuing into the final presentation. This loss of initial successes is substantially larger than under fixed-order repetition, even for highly repeatable Jev reproductions. Nor does Jev show a general advantage over the LLM baselines in preserving correct choices. Aggregation offers limited protection: voting yields modest gains for Jev, while even unanimous answers are frequently wrong. These results expose a gap between repeatable decisions and decisions that remain correct across equivalent presentations, showing why stable accuracy alone cannot establish resistance to selection bias.