Biomedical reranking improves clinical evidence retrieval but not answers
Evaluating Biomedical Reranking for LLM-Based Question Answering over Longitudinal Clinical Notes
Computation and LanguageEmerging Technologies
Summary
Finding specific information in long and complex medical records is hard because important details can be scattered or copied in different ways. The authors tested a new technique that reorders search results to put the most relevant medical evidence higher up. They found this method helped locate the right pieces of the medical records more often, but it only slightly improved the correctness of answers generated from that evidence. So, better searching doesn’t always lead to perfectly correct answers.
What this means in practice
- •For hospital data teams: Enhance retrieval accuracy within electronic health records to surface relevant clinical notes for patient-specific queries in complex longitudinal data.
- •For clinical ai developers: Improve ranking of clinical evidence retrieved by AI systems to support question answering over patient records despite heterogeneous note structures.
Authors
Maryam Shahbaz Ali, Laura B. Strachan, Caitlin Sherman, Mark Kovler, Eleanor Mackey, Syed Muhammad Anwar
Abstract
Patient-specific clinical question answering requires locating the right evidence within long, heterogeneous longitudinal clinical records in which relevant facts may be scattered across encounters, repeated in copied-forward notes, or expressed using different clinical terminology. We evaluated whether biomedical reranking can improve evidence selection and downstream answer quality in a locally deployed retrieval-augmented generation pipeline for longitudinal clinical notes. The pipeline combines PubMedBERT dense retrieval, BM25 lexical retrieval, weighted reciprocal-rank fusion, and MedCPT cross-encoder reranking. Across 1,000 open- and closed-ended question-answer pairs from a cohort of 200 bariatric surgery patients, reranking increased exact source-chunk retrieval within the top 10 items, Hit@10 from 46.6% to 60.6% and mean reciprocal rank from 0.2371 to 0.3252. With Qwen3-8B generation, local judge-assessed answer correctness increased from 44.8% to 48.6%. These results show that biomedical reranking can improve the placement of relevant clinical evidence within a limited context window, although gains in retrieval do not translate proportionally into gains in answer correctness.