Evaluating reasoning chains must consider answer correctness bias
Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation
Artificial Intelligence
Summary
When checking if AI reasoning steps (chain-of-thought) are reliable, focusing only on questions the AI answers correctly changes what we measure. The authors explain using a mathematical model how filtering by correct answers can mix up causes like task difficulty and misleading shortcuts. They analyze past studies and test their ideas on math problems, showing this filtering creates noticeable differences in faithfulness scores. They propose that if one can observe compliance with reasoning for every case, then evaluating all attempts directly measures true faithfulness without filtering.
What this means in practice
- •For machine learning engineers: Improve AI reasoning system evaluations by adjusting for biases introduced when only correct answers are analyzed.
- •For data quality teams: Design evaluation protocols that include all reasoning attempts to avoid selecting biased subsets and better measure AI output reliability.
Authors
Yueer Pan, Yuexi Pan
Abstract
Restricting chain-of-thought (CoT) faithfulness evaluation to correctly answered instances changes the population being measured. We study this selection step in a seven-variable structural causal model and derive the exact observed gap $Δ= \operatorname{Cov}(A, C)/P(C=1)$. The model distinguishes difficulty confounding from shortcut-induced collider bias and states the conditional-independence assumption needed to express the gap through latent response propensities. A protocol review of 18 papers distinguishes answer-correctness restrictions from agreement-based selection, benchmark construction, and training-data filtering. We evaluate four sub-ceiling deployments on arithmetic tasks and observe positive selection gaps in all four. The primary GLM-5.3 run yields a +7.4 percentage-point gap ($n = 1000$, $p < 0.0001$), and pooled Cochran-Mantel-Haenszel analysis gives OR = 1.70 ($p < 10^{-6}$). An assumption-free covariance decomposition assigns 42.8% of the GLM-5.3 gap to between-stratum variation in interventionable-operation count, leaving the majority unattributed. When counterfactual compliance is observable for every instance, evaluating all traces directly estimates population faithfulness without correctness filtering.