Rag-nimbf enables systematic testing of retrieval-augmented generation setups
RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis
Software EngineeringArtificial Intelligence
Summary
Large language models combined with retrieval of external information can give more accurate answers but involve many settings to adjust. The authors introduce RAG-NIMBF, a tool that attaches to existing systems without changing them, letting developers test different choices systematically. It measures how well the retrieved information supports the model’s answers and tracks experiments end to end. This helps find better settings like balancing relevance and diversity in retrieval and giving clearer prompts to improve answer faithfulness.
What this means in practice
- •For ai system engineers: Evaluate how different retrieval and prompting settings impact answer quality without disrupting live language model applications.
- •For customer support platform teams: Improve automated response accuracy by benchmarking retrieval strategies and prompt designs integrated into existing chatbot pipelines.
Authors
Niclas Hart, Nicolas Weeger
Abstract
Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime indicators. This study validates the framework across multiple generation models on the SQuAD dataset. This process reveals non-obvious interactions and demonstrates how the experiments are tracked. The findings indicate that the retrieval balance of relevance and diversity substantially enhances the relevance and coverage of retrieved context in comparison to a conventional similarity search, with MMR raising nDCG@5 by +4 to +11 percentage points across the swept models. More detailed prompt instructions meaningfully increase how faithfully generated answers stick to the retrieved evidence, especially for capable and reasoning models (+10 to +13pp on Qwen3.6-27B, GPT-4o-mini and Qwen3.5-think, but negligible on the 20B generator). Overall, RAG-NIMBF transforms the configuration space of RAG into reproducible, comparable experiments. By attaching to existing production pipelines in a non-invasive way, it enables practitioners to systematically evaluate and improve deployed systems without disrupting existing workflows. This replaces ad hoc choices with evidence-based decisions.