Large language models extract structured policy data at scale
From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
Computation and LanguageArtificial Intelligence
Summary
Different government policies can be hard to keep track of because there are so many and they vary a lot. The authors show how large language models (LLMs) can act like automated survey takers, reading long policy documents and pulling out key information in a structured way. They compare this automatic extraction to answers from human experts and find that LLMs are very good at identifying specific policy details but sometimes add more explanations when writing freely. This approach could help monitor policies more efficiently, though humans still need to check the results for accuracy.
What this means in practice
- •For government policy teams: Automatically create structured reports from diverse policy documents to improve monitoring across countries.
- •For international organizations: Scale up data collection efforts on policy measures across nations by supplementing surveys with AI-extracted information.
Authors
Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani
Abstract
Science, technology, and innovation policies are crucial for competitiveness, yet their diversity and scale make them difficult to map and monitor consistently. Existing approaches rely heavily on manual survey efforts, which are costly and challenging to scale across countries. Large language models (LLMs) enable new possibilities for extracting and structuring information from long and unstructured policy documents. This paper presents an application of LLMs as "AI respondents" for generating structured survey responses from policy texts. We develop a data extraction pipeline based on long-context in-context learning to map information from public web sources into predefined survey categories, including policy instruments, target groups, and thematic areas. The pipeline integrates a validation step using a secondary LLM to assess relevance and evidence, alongside comparisons with human-provided responses. Using a multi-country dataset, we evaluate the alignment between LLM-generated and human-generated outputs through overlap measures and cross-validation. Results show that LLMs achieve high agreement for structured indicators (84-95%), while differences remain in free-text fields, where models tend to provide more detailed procedural descriptions. These findings highlight the potential of hybrid human-AI workflows for policy monitoring, improving both efficiency and scalability while maintaining the need for human validation and contextual interpretation.