CIVI framework finds search agents often fail on government information
CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information
Artificial Intelligence
Summary
Computers that help people find government information sometimes give wrong answers, which can cause serious problems. The authors created CIVI, a way to check where these search helpers are making mistakes, especially across different levels of government and topics. They tested ten advanced search tools and found none were as good as careful humans. Their method, called ARISE, shows that most errors happen because the tools don’t find the right information, not because they forget what they know.
Large language modelsSearch agentsCivic informationGovernment sourcesInformation retrievalBenchmarkingFailure diagnosisParametric knowledge
Authors
Dingying Liu, Yunshun Zhong, Wentao Zhang, Yiyuan Li
Abstract
Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models' parametric knowledge.