Trie-based Experiment Plans for Efficient IR Pipeline Experiments
2026-07-01 • Information Retrieval
Information Retrieval
AI summaryⓘ
The authors explain that search engines often work in steps, where each step improves the list of results before showing them to users or feeding them to language models. They focus on making experiments that compare these step-by-step pipelines more efficient, using a trie data structure to organize the tests better. By doing this, their method reduced the time needed for experiments by about 26% in one test case. They also studied how students used these improved experimental plans.
search enginescascading pipelinesretrievalrankingPyTerriertrie data structureBM25MonoT5DuoT5MSMARCO
Authors
Irene Anu, Craig Macdonald
Abstract
Search engines are often formulated as cascading pipelines, where successive stages combine the results of different retrievers, and iteratively refine the ranking of candidate documents to obtain a final ranking, which can be presented to a user, or provided as context to an LLM. Such pipelines can be complex to evaluate in an end-to-end manner, necessitating measurement of Recall of early stages, and Precision of later stages, which are often interchangeable. PyTerrier is ideal for building and evaluating cascading retrieval pipelines, due to its declarative nature for pipeline construction and wide ecosystem of retrievers and rerankers. However, comparative evaluation of pipelines can be expensive due to repeated components. In this work, we describe the use of a trie data structure to formulate an experiment plan for comparative pipeline experiments that enhances experiment efficiency compared to a sequential "linear" plan. Empirically, on a demonstration experiment involving BM25, MonoT5 and DuoT5 on MSMARCO v2, we observe a 26% reduction in experiment duration. Finally, we report on a user study of undergraduate and postgraduate research students' use of the experiment plans.