Thinking in language models fixes some biases but causes more
Does Thinking Help Fairness? Reasoning Tokens Resolve Some Biases but Create More
Artificial Intelligence
Summary
This paper looks at whether having language models 'think' before answering helps reduce unfairness in their decisions. The authors found that while thinking does fix some mistakes related to bias, it also creates many new errors that were not there before. In fact, the new mistakes happen about five times more often than the ones fixed. They studied how bias changes during the thinking process and found it tends to build up instead of going away. This means thinking can both help and hurt fairness in these models.
What this means in practice
- •For machine learning engineers: Evaluate fairness effects of adding explicit reasoning steps in AI models for high-stakes decision tasks to balance bias resolution and creation.
- •For ai product managers: Inform design choices about incorporating reasoning tokens in AI systems where fairness is critical, understanding potential trade-offs.
Authors
Deng Pan, Joe Germino, Yihong Ma, Elizabeth Daly, Nuno Moniz, Ting Hua, Nitesh Chawla
Abstract
Thinking in reasoning language models (RLMs) has been subject to debate on whether it resolves or amplifies bias. Prior works have shown competing conclusions in both directions. Using a within-model thinking-vs.-non-thinking ablation across QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and Qwen3-32B on three high-stakes decision tasks (Adult, COMPAS, Credit), we show that thinking has an asymmetric dual effect on counterfactual fairness: it both resolves counterfactual flips produced by the non-thinking baseline and creates new flips at near-saturating model confidence. In all nine (model, dataset) combinations, the created flips outnumber the resolved flips by roughly 5 times. To explain the effect, we treat the thinking trace itself as a measurable site of fairness change and study it through two dynamic instruments: 1) We propose Counterfactual Depth Probability Gap (CDPG) to track bias evolution along thinking depth, and observe that bias propagates and amplifies with thinking. 2) We also formulate the Bias Transition Matrix (BTM) to show how predictions of counterfactual pairs change from non-thinking to thinking, and find that the asymmetric dual effect originates in the pair-state joint transition.