Instruction following retrieval improves with natural hard negatives
Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval
Artificial Intelligence
Summary
Finding relevant documents based on specific instructions is hard because there are few ways to teach machines to distinguish subtle differences. The authors found that using original documents as tricky negative examples, rather than just generating synthetic ones, helps models learn better. They created an automated process to find these natural hard negatives and used it to produce a new training set. Models trained on this data performed better at following detailed search instructions, even competing with much larger systems.
What this means in practice
- •For search engine teams: Build retrievers that better understand detailed user instructions to rank relevant documents more accurately.
- •For enterprise search developers: Improve internal document retrieval systems by incorporating training data with natural hard negatives to refine instruction-based search.
Authors
Chaolang Li, Jingyu Li, Zhaocheng Du, Xuyang Xie, Gang Wang, Pengwen Dai
Abstract
Instruction-following information retrieval (IFIR) requires retrievers to rank documents by both a query and its instance-specific instructions, yet instruction-conditioned contrastive supervision remains scarce. Existing synthetic methods typically fix a positive passage and generate additional negatives, overlooking the hard-negative potential of the original positive. We propose Natural Hard-Negative Mining (NHNM), which promotes convertible negatives under query-compatible instructions and repurposes original positives as natural instruction hard negatives. Its automated Candidate Cleaning, Reversal Construction, and Geometry Filtering pipeline produces compact, high-contrast supervision aligned with the pretrained retrieval space. Applied to MSMARCO, NHNM yields NHNM-MARCO, an IFIR training set of nearly 7K instances. On InstructIR and FollowIR, models with 568M to 4B parameters fine-tuned on NHNM-MARCO achieve strong instruction-following performance and match or surpass substantially larger retrievers in several settings. Controlled analyses and ablations attribute these gains to natural hard-negative supervision rather than model scale or training-data volume.