Listwise Cross-Encoder Fine-Tuning vs. Agentic Instruction Tuning for LLM Rerankers: A Systematic Study in Medical Procedure Reranking
2026-08-10 • Information Retrieval
Information RetrievalComputation and Language
AI summaryⓘ
The authors studied how to improve the ordering of medical procedures when patients search for insurance information, which is hard because patients and doctors use very different words. They compared two approaches: a small, specialized model fine-tuned to rank lists, and a much larger model that uses instructions refined by another AI (GPT-4). Surprisingly, the smaller model performed better despite having far fewer parameters. The authors also shared details about their dataset, methods, and practical considerations for real-world use.
medical procedure rerankinghealth insurance information retrievalcross-encoderlistwise learning-to-rankNDCG@3Spearman correlationGPT-4instruction-based rerankerlayer freezingparameter efficiency
Authors
Matan Fainzilber, Shlomit Plavner
Abstract
Reranking medical procedures against patient queries is a critical component of health insurance information retrieval, complicated by a substantial lexical gap between patient language and clinical nomenclature. We present a systematic comparison of two reranking paradigms for this production task: (1) small cross-encoders (MedCPT, MiniLM-L12) fine-tuned with listwise learning-to-rank objectives across layer freezing configurations, and (2) Qwen3-Reranker-4B, a 4B-parameter instruction reranker whose prompt is iteratively refined via an agentic optimization loop driven by GPT-4.1. On a purpose-built dataset of 2,647 queries across 708 insurance services, we find that a 109M-parameter cross-encoder fine-tuned with ListNet outperforms the 4B-parameter model by 2.6 percentage points on NDCG@3 and 13.3 points on Spearman correlation - at 37x fewer parameters. We report practical findings, a scalable LLM based dataset construction pipeline, and deployment trade-offs relevant to production reranking systems. We release our code and a sample dataset to support reproducibility and adaptation to other domains.