Evidence-aware approach improves retrieval of breast lesion information

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

Computation and LanguageArtificial Intelligence

Summary

Finding the right information in medical articles about breast lesions can be tricky because it depends on how the facts are marked up. The authors looked at how adding more detailed annotations of evidence affected searching accuracy in a small collection of documents. They combined different search methods and showed their best setup almost always found useful evidence near the top of the search results. Adding more evidence markers allowed the system to recognize more helpful text without changing the order of retrieved results. This means how you label information changes how well search scores reflect actual helpfulness.

What this means in practice

  • For clinical informatics teams: Improve clinical literature search systems by incorporating evidence-aware annotations to better identify and rank relevant breast lesion management information.
  • For medical ai developers: Design retrieval-augmented generation tools that account for annotation coverage to enhance accuracy in clinical question answering over breast lesion documents.

Tested on one dataset.

Authors

Miguel Mochizuki Silva, Samuel Ismael, Mateus C. Dias, Thaís Gaudêncio, Yuri Malheiros, Barufaldi

Abstract

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrieval-augmented generation (RAG) system over breast lesion management literature, addressing the dependence of retrieval scores on annotation coverage. Methods: An 18-document corpus and an 18-query development benchmark, with one Yes/No or WH-question per document, were used. Retrieval combined dense embeddings, BM25, and cross-encoder reranking. Chunk sizes of 128 to 1024 tokens, overlaps of 0 to 20%, and four retriever configurations were evaluated. Relevance was defined by evidence spans mapped to all containing chunks, treating overlapping chunks with equivalent evidence as relevant. Retrieval was evaluated using R-Precision, Recall@K, MRR, and nDCG@K. Separately, generation was assessed on a 512-token index at K=1 to 5 using DeepEval claim-level metrics, with deepseek-v4-pro as the LLM judge, and BLEU/ROUGE metrics. Results: The selected hybrid retriever, using 384-token chunks and 20% overlap, retrieved accepted evidence in the top three results for 17 of 18 queries. Recall@3, R-Precision, MRR@3, and nDCG@3 each reached 0.94. Ground-truth curation increased evidence spans from 25 to 37 and accepted chunk targets from 36 to 51 without changing retrieval rankings. Conclusion: Cross-encoder reranking achieved the highest retrieval scores. Expanded relevance annotations increased recognized evidence without changing rankings, highlighting annotation sensitivity. The small development set and retrieval-informed curation require independent validation, including radiologist assessment of evidence sufficiency.
View PDFOpen arXiv