Retrieval systems perform differently on synthetic versus real user questions
Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation
Information Retrieval
Summary
Evaluations of retrieval systems often use artificially created questions based on the documents they search over, but these questions are longer and cover more topics evenly than real user queries. The authors compared synthetic questions to actual questions students asked about a university faculty system. They found that real queries were shorter, focused on fewer topics, and retrieval systems that performed well on synthetic questions did worse on real ones. This suggests using both synthetic and real queries helps understand system strengths and weaknesses better.
What this means in practice
- •For enterprise search engineers: Design system evaluations that include real user query data to ensure practical retrieval performance under typical conditions.
- •For university it support teams: Tailor retrieval system configurations to authentic student queries for more responsive and relevant university information services.
Tested on one dataset.
Authors
Filip J. Kucia, Barbara M. Gawlik
Abstract
RAG systems are routinely evaluated using synthetic question sets generated from the target document corpus. While this practice provides a useful check on overall retrieval capability, relying exclusively on synthetic benchmarks can mislead under distribution shift and overstate deployment readiness. Synthetic generation spreads questions evenly across the corpus, formulating long, detailed queries; real users put most of their traffic on a few administrative and procedural topics in short queries, while also asking about matters the generator never covers at all. We demonstrate this gap on a university faculty information system, comparing 1,851 synthetic questions generated via Gemini Notebook against 322 authentic queries collected via a student survey. The synthetic and authentic query sets differ significantly: authentic queries average 6.8 words versus 15.7 for the synthetic ones, and draw from only 53 unique sources compared to 165. Consequently, configurations that appear highly effective on synthetic benchmarks experience a substantial performance drop on authentic queries. Importantly, optimizing on synthetic queries selected a higher-latency hybrid retriever. In our setting the sparse retrieval component benefited long synthetic questions but not short authentic ones, costing up to $8\times$ the latency of the fastest configuration we tested. We propose treating synthetic and authentic query sets as complementary extremes of the query-quality spectrum: synthetic data verifies maximum retrieval capacity under idealized conditions, while authentic queries test system robustness to the imprecise, underspecified inputs of real users.