Curated retrieval versus open web search in public AI information services: a coverage-trust trade-off
2026-07-06 • Computers and Society
Computers and SocietyComputation and LanguageInformation Retrieval
AI summaryⓘ
The authors studied an AI service called Evrópuvefur, which answers questions about the European Union using either a curated database or live web search. They found that answers based on web search often included sources that experts flagged as untrustworthy or irrelevant, while curated sources were more reliable but covered fewer topics. The AI sometimes avoided answering when it lacked good information from curated sources. The study highlights that trustworthiness of sources is an important but overlooked part of AI answer quality.
large language modelsretrieval-augmented generationcurated knowledge baseweb searchsource trustworthinessexpert evaluationinformation qualityprompt engineering
Authors
Hafsteinn Einarsson, Hafsteinn Birgir Einarsson, Jón Gunnar Ólafsson, Jón Gunnar Þorsteinsson
Abstract
Public institutions increasingly use large language models (LLMs) to answer citizens' questions, often pairing a curated knowledge base with live web search, yet whether the sources behind these answers can be trusted has received little empirical scrutiny. We report a pre-launch expert evaluation of Evrópuvefur, an independent, government-funded service run by the University of Iceland that answers questions about the European Union, conducted as Iceland prepared for its referendum of 29 August 2026 on whether to resume EU accession talks. Five domain experts produced 551 evaluations of 449 AI-generated answers, scoring each against a seven-criterion quality rubric and, separately, flagging individual cited sources. We compared two retrieval paths: a curated local corpus (RAG) and open web search. In more than a third of the reviewed web-search answers (35%, 65 of 187), at least one cited source was flagged, almost always as untrustworthy or irrelevant; curated sources were flagged far less often and only for being out of date. Web search answered more questions, but at the cost of source quality; the curated corpus was trustworthy yet limited in coverage, and the model declined to respond when it fell short. The citation mix also passed over strong sources: across all 287 web-search answers, the system never cited RÚV, the public broadcaster and the country's most widely used news source. A companion prompt ablation shows how weak prompt-level steering is: a trusted-domain list in the system prompt raised the share of citations to listed domains only from 12% to 21%. Fluency and topical fit did not predict source trustworthiness. We argue that source trustworthiness is a measurable yet largely invisible dimension of information quality in public AI services, and we discuss transparency-oriented responses and their trade-offs.