government data teams

Abstract: Public bodies increasingly rely on AI-produced evidence, including classifications, summaries and assessments. These outputs can inform policy decisions and service delivery, making their reliability a matter of public accountability. Establishing whether this evidence can be relied upon requires testing its outputs and examining how organisations decide whether those tests are sufficient. We combine technical evaluation with analysis of AI governance requirements, informed by measurement theory and management-based regulation. We compare eight GPT-4o configurations classifying 1,200 UK Prime Minister's Questions into 21 policy topics, against human-coded labels and across unchanged repeats. Accuracy ranges from 68.2% to 70.3%, yet 24.3% of questions receive different labels, with changes in topic shares and rankings. Configurations agree more with one another than with human coding: agreement does not guarantee correctness. Under the implemented score-extraction procedure, the proportion flagged for human review ranges from 0.9% to 41.0%. We analyse 37 UK and EU policy documents. Both sets expect testing and oversight, but reporting expectations do not consistently cover changing answers, shared errors, selective evaluation, substantive conclusions or review routing. Some UK guidance explicitly addresses stability. The UK sample emphasises public bodies checking their own use; the EU places substantial testing duties on providers while leaving them important choices about measures and acceptance thresholds. Without appropriate checks, evidence could misrepresent policy issues, redirect attention or resources, and leave errors undetected. Public policy should require organisations to justify reliance through checks beyond aggregate accuracy, clear responsibility, access to expertise and scrutiny of results.

Abstract: We compare how often two released language models give outdated numeric values from Norwegian statutes and regulations: NbAiLab/borealis2-26b-a4b-preview (Borealis 2), which the National Library of Norway further trained from google/gemma-4-26B-A4B-it (Gemma 4), and Gemma 4 itself. The study was preregistered (doi:10.5281/zenodo.23858965) and uses the closed norregel benchmark, scored deterministically against a value history. On the 411 of 503 primary items where both models gave an interpretable answer, Borealis 2 alone was outdated on 10 items and Gemma 4 alone on 11 (d = -0.2 percentage points, 95% cluster bootstrap interval [-2.7, +2.5], exact one-sided McNemar p = 0.668). By the preregistered rule this is no effect of interest: the whole interval lies within the smallest effect of interest of +/-10 percentage points. Among the secondary measures, only the share of answers giving a value that took effect after the question date differs after Holm correction; Gemma 4 gave such answers more often. An exploratory comparison of two smaller Borealis models had no discordant pairs and is not informative. The design compares released models and does not isolate the effect of any part of their training.

Abstract: Gaps, skips, and mistakes are the normal state of a historical register: rows go missing, tables cross page breaks, columns slip under wrong names, and column sums disagree with the corresponding row entries. Readers read digits correctly but lose the table structure. On seven clean panels of election returns, the best open-source reader reproduces 99.0\% of the digits but files only 83.7\% of the figures in the correct cells and gets 11 of 30 columns correct. TableVision treats table structure recovery as the problem. On three degraded 1888 pages, Amazon Textract files only 1 of 1494 figures correctly; TableVision correctly files 99.2\%, and all 51 columns its exact-total check releases are correct. A dynamic program cuts a volume into blocks by how many columns sum to their last line; witnesses read the cells; the gate classifies each column as exactly reconciling, flagged as faithful to a page whose arithmetic is wrong, or withheld. A theory of the sum check says which errors it cannot see and why its optimal block is about $θ^{-1/2}$ rows at defect rate $θ$. On 2201 columns of Wisconsin electoral data, the gate accepts 2042, flags 118, and withholds 41; failure rises with column length $R$ as $1-(1-θ)^R$ at 1.1 defects per 1000 cells in general elections; official returns confirm 933 of 934 testable accepted sums and all 35 testable flagged totals; and the deployment added 491\,359 gated precinct rows to a 9.3-million-row dataset. One implication is that the design of the verification block is critical.

Abstract: This paper links all individuals who spoke in the German Bundestag between 1949 and 2021 to Wikidata, creating a longitudinal dataset that connects parliamentary speech transcripts with structured biographical metadata. We evaluate the coverage, composition, and potential biases of the retrieved properties and statements, with particular attention to gender, professional background, historical legacies, and transnational dimensions such as place of birth, languages, and foreign awards. The results demonstrate both the analytical potential of combining GermaParl with Wikidata and the importance of critically assessing uneven metadata coverage in open knowledge graphs.

Abstract: Harvard Dataverse hosts over 150,000 research datasets, but the geographic information those datasets carry is entered as free text by depositors and has never been assembled into a searchable structure. We construct a knowledge graph from the repository's public data and metadata, organizing 102,650 datasets within a 215,985-node network of 528,003 edges linking datasets to keywords, publications, subjects, journals, and locations. Of those datasets, 43,991 (42.9 percent) carry at least one geospatial field, geographic coverage, geographic unit, or a bounding box and 96.9 percent of all nodes sit in a single connected component, so datasets remain reachable from one another even when their geospatial metadata share nothing in common. A conservative keyword search identifies 7,654 geospatially tagged datasets (17.4 percent) as directly policy-relevant, with elections and legislatures the largest cluster, followed by government administration, health policy, transportation, and education. Five datasets illustrate how this metadata behaves across policy domains and spatial scales, and an extended use case shows how community language models, stance detection with geographic aggregation, and partisan language bridging tools can attach discourse to place. The central obstacle is place resolution: the same location appears as many disconnected nodes. We argue that the graph provides a concrete setting for developing AI-driven metadata enrichment and entity resolution, and we document its coverage skew toward American, city-level data.