Search agents struggle to provide accurate civic information across governments
CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information
Artificial Intelligence
Summary
Many computer programs called search agents help people find government information, but mistakes can cause real problems. The authors created a way to check why these programs fail, looking at different countries and government levels. They found that most errors happen because the search part misses important information, not due to the program’s own knowledge. This helps show where improvements are needed to make these tools more reliable.
What this means in practice
- •For public sector software teams: Assess and improve government information search tools by identifying where failures occur in retrieval versus knowledge gaps.
- •For legal compliance teams: Monitor how well AI tools cite authoritative government sources to ensure accurate civic guidance.
Authors
Dingying Liu, Yunshun Zhong, Wentao Zhang, Yiyuan Li
Abstract
Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models' parametric knowledge.