Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking
2026-08-31 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied how models rank documents one by one and found that models using detailed reasoning steps (Chain-of-Thought) don't rank as well as simpler direct scoring models. They tested different training methods to fix this but found that while some improvements were possible, the ranking difference stayed the same. Their work suggests that trying to turn complex relevance ideas into text limits how well these models can rank documents. This limitation seems hard to fix with current standard approaches.
pointwise rankingChain-of-Thoughtdirect scoringreinforcement learningclassification accuracyranking gapmodel scalingfine-grained supervisionarchitectural decouplingrelevance semantics
Authors
Xiaoyang Chen, Jie Liu, Haijin Liang, Haibo Shi, Jin Ma, Ben He, Yingfei Sun, Dezhi Ye
Abstract
In pointwise document reranking, Chain-of-Thought models typically underperform direct scoring models. While existing diagnostics attribute this to inferior classification, score polarization, or calibration breakdown, whether targeted training can bridge this gap remains unclear. Our empirical study first confirms that this gap is stable across scales up to 32B parameters, ruling out model and data capacity confounders. We then apply stress tests utilizing reinforcement learning, fine-grained supervision, and architectural decoupling to explicitly repair these deviations. Although these interventions improve classification accuracy and absolute scores, the relative ranking gap persists. These findings suggest that, within the pointwise scoring paradigm, routing continuous relevance semantics through discrete text constrains ranking signal resolution, revealing a bottleneck that is stable and difficult to overcome under current standard methods, rather than an easily resolvable training bias.