Calibration of biomedical question answering evaluation improves reliability
Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals
Computation and LanguageMachine Learning
Summary
Checking how good biomedical question answering systems are is hard because automatic scores often don’t match what humans think. The authors studied how using a small amount of human preferences can adjust these automatic evaluation metrics to make them more accurate. They found that their adjusted scores did a better job than just using all metrics equally or picking one metric for all situations. This helps developers know which system improvements really matter.
What this means in practice
- •For biomedical ai developers: Improve tuning of biomedical QA systems by using calibrated evaluation metrics informed by a few human preferences for more reliable assessment.
- •For medical informatics teams: Guide selection and improvement of biomedical QA tools with evaluation metrics better aligned to expert judgments.
Authors
Shakila Mahjabin Tonni, Xiang Dai, Sarvnaz Karimi, Xiaomei Li, Samuel C. Andrew, Stephen Wan
Abstract
Retrieval-Augmented Generation (RAG) systems are increasingly used for biomedical Question Answering (QA). However, evaluating their effectiveness remains challenging due to the lack of reliable reference resources and the frequent misalignment between automatic evaluation metrics and human judgments. One practical question is how evaluation results should be used to guide development decisions, especially when they correlate weakly with human judgments and may even lead to different conclusions. We explore how to use a small amount of human preference signals to calibrate direct assessment evaluation metrics, and show that calibrated metrics consistently outperform baselines that either treat all metrics equally or rely on a single best metric across multiple datasets.