Papers for

public sector software teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Search agents struggle to provide accurate civic information across governments

CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information

Abstract: Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models' parametric knowledge.

Tue 8 SeptArtificial Intelligence
The gist
Many computer programs called search agents help people find government information, but mistakes can cause real problems. The authors created a way to check why these programs fail, looking at different countries and government levels. They found that most errors happen because the search part misses important information, not due to the program’s own knowledge. This helps show where improvements are needed to make these tools more reliable.
Open → 2609.08094v1