Embedding searches find more trial evidence than label searches
Nothing Left Behind: Finding Evidence from a Large Randomized Evaluation Corpus
Computation and Language
Summary
Finding every relevant piece of information in large collections is hard because traditional label-based searches miss many important items and catch too many unrelated ones. The authors compared this old way with a newer method that first finds candidates based on meaning using text embeddings, then uses a language model to keep only what matters. Tested on a large set of randomized trials, the new method found much more relevant evidence than label searches, nearly doubling the recall. They suggest using a two-step approach: gather lots of possible results broadly, then carefully filter them by relevance.
What this means in practice
- •For health data teams: Improve systematic reviews by retrieving more relevant trial evidence through combined embedding and filtering methods.
- •For legal discovery teams: Enhance document search accuracy by first broadly finding meaning-based candidates, then filtering those using a language model.
Authors
Giuliano Martinelli, Abelardo Carlos Martinez Lorenzo, Satvik Garg, Linxi Wang, Riccardo Orlando, Piriyakorn Piriyatamwong, Samuel Paul Fraiberger
Abstract
Many AI systems need to find every relevant items in a large collection and then organize what they found. The standard way to do this is to tag each item with labels from a curated vocabulary and search on those labels. Those labels are useful for organizing results, but searching on them misses items that do not fit a label and returns too much when labels are broad. We compare this approach with an embedding-based approach: find candidates by meaning using text embeddings, then have a language model keep only the relevant ones. In a corpus of 7,354 randomized trials, label search found 45% of the evidence a systematic review would include, while the new approach found 92%, at a cost per query that rises with search depth. These results support a two-stage architecture that combines broad candidate retrieval with query-specific evidence filtering.