Decision models do not improve reranking over standard fine-tuning
System One models Are Not Better Rerankers
Computation and LanguageInformation Retrieval
Summary
The paper compares two ways to improve search ranking models: decision fine-tuning, which calibrates probability answers, and typical reranker fine-tuning. The authors find that decision fine-tuning does not make models better at reranking results compared to standard approaches. This holds true across different model sizes and testing collections. They also show that better calibration of decisions does not compensate for poorer ranking quality. Overall, fine-tuning a reranker remains the best choice for improving search rankings.
What this means in practice
- •For search engine developers: Choose standard reranker fine-tuning over decision fine-tuning for improving search result rankings in large language models up to 9B parameters.
- •For digital library engineers: Use conventional reranker fine-tuning to enhance document retrieval effectiveness on established datasets like TREC DL and BEIR.
Authors
Michael Dinzinger, Kanishka Ghosh Dastidar, Michael Granitzer
Abstract
System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.