Biomedical sentence encoders improve typed decision models accuracy
From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders
Computation and LanguageArtificial Intelligence
Summary
Deciding answers to complicated questions from biomedical texts is challenging. The authors studied if models trained to find related sentences could be adapted to answer specific biomedical questions efficiently. They found that starting from retrieval-trained models helps some types of question-answering models but not others. They also discovered an improved way to adjust these models for better decision confidence. Their work includes new tools and a large dataset to help build better biomedical question-answering systems.
What this means in practice
- •For biomedical data teams: Build systems that answer structured biomedical questions with calibrated confidence from large medical text collections.
- •For clinical informatics developers: Improve clinical decision support tools by integrating calibrated typed decision models built on biomedical text encoders.
Authors
Pritam Deka
Abstract
Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask whether biomedical sentence encoders trained for retrieval are good starting points for such models. We present SBERT2S1, which converts Sentence-Transformers encoders into bi-encoder, cross-head (C) and prior-fused residual (PFR) decision models, together with BIODECIDE, a biomedical typed-decision suite, and MEDLINE-S1, 243k training decisions derived from NLM indexing. Across six parent-retriever pairs, retrieval training improves zero-shot matching of content-bearing options. After fine-tuning, its effect depends on the head: across five pairs and three training-set sizes, retrieval training significantly helps PFR, which keeps the retrieval prior, in 10 of 15 comparisons, but helps C in one and hurts it in five. A matched grid of two heads and five training objectives shows that C outperforms PFR under every objective, and that the released RLCD recipe of open System One models trails cross-entropy by 2.5-3.0 points. The deficit stems mainly from its reward normalisation, which inflates the noisy score-function term 3.6-15-fold; an unbiased leave-one-out estimator recovers most of the gap. After temperature scaling, no objective is clearly better calibrated than cross-entropy. We release the code, the MEDLINE-S1 labels and a model.