Techniques for retrieval augmented generation do not always add up together
Retrieval Augmented Generation Compositionality
Computation and LanguageArtificial Intelligence
Summary
Combining different methods to improve large language models by giving them external information doesn’t always work as expected. The researchers found that methods that work well alone may not work well when combined. They also discovered that what works best for small collections of documents might not be best for very large ones. This means testing parts of these systems together in realistic settings is important before deciding which methods to use.
What this means in practice
- •For search engine developers: Design retrieval pipelines that are tested as a whole to get more reliable and accurate search results from large document collections.
- •For enterprise knowledge management teams: Improve information retrieval systems by choosing retrieval and verification techniques that work robustly at scale across large corporate data repositories.
Authors
Ido Levy, Segev Shlomov, Leshem Choshen
Abstract
Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.