Training with negative passages improves answer ranking after calibration
Training Negatives Still Matter After Calibrating Answerability Readouts
Computation and Language
Summary
Sometimes, computer programs try to find if a passage answers a question or not. This paper shows that training these programs using passages that do not contain answers (negatives) helps them better decide and rank answers even after their predictions are adjusted (calibrated). The researchers compared training with and without these negative passages and found that keeping negatives leads to better overall performance, especially in ranking answers. However, they also note that this benefit is not the same in every situation or dataset.
What this means in practice
- •For question answering system developers: Improve answer ranking and decision accuracy by training models with negative passages, refining output calibration without changing model architecture.
- •For search engine engineers: Enhance passage selection for question answering features by incorporating negative passage training for better differentiation between answerable and unanswerable content.
Authors
Jinchen Zhang, Zheyuan Liu, Yihan Zhu, Meng Jiang
Abstract
Answerability readouts predict whether a passage supplies an answer, but whether calibration can compensate for removing negative passages during training remains unclear. We present a controlled study of how training-negative construction affects calibrated prediction, ranking, and downstream answer retention. We compare native training, which retains negative passages, with empty training, which removes them, holding questions, labels, positive examples, and model weights fixed. Both are calibrated on the same labeled native examples. On SQuAD 2.0 and RepLiQA, native-trained logistic readouts at selected depths achieve lower Brier loss for Qwen2.5-7B-Instruct and Gemma-2-9B-it, with or without output scores. Native fusion also outperforms calibrated output alone. Post-hoc AUROC analysis shows better ranking under native training, beyond what threshold adjustment alone can explain. On the official SQuAD 2.0 development set, native readout policies improve F1 over empty policies but retain more answers to unanswerable questions. The advantage varies by estimator and depth, while RepLiQA's text shortcuts limit claims about contextual understanding. Unfinished generations prevent complete answer-level evaluation, although worst-case bounds preserve the native-empty F1 gains. Our findings show that training negatives affect ranking and downstream decisions even after common calibration, without establishing a universal advantage for native training.