AI reliability checks needed for public policy evidence use
Beyond accuracy: what AI assurance needs to establish in public services
Computers and SocietyComputation and Language
Summary
Government agencies use AI tools to help sort and understand political topics, but these tools don’t always agree or give consistent answers. The authors tested different AI setups and found that even when accuracy rates look okay, the AI often changes its answers or disagrees with human judgment. They also looked at UK and EU rules and found that current policies don’t fully require checks for AI answer changes or shared mistakes. The paper suggests governments should demand better AI checks to make sure AI results can be trusted in public decision-making.
What this means in practice
- •For government data teams: Design testing protocols that go beyond accuracy to detect AI answer changes and shared errors when categorizing policy topics for oversight purposes.
- •For ai compliance officers: Create frameworks to ensure AI outputs used in public service meet governance standards requiring stability, detailed error analysis, and human review triggers.
Authors
Louise Yorke
Abstract
Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.