Summary
It can be hard to find where specific data sources are mentioned in reports about forced displacement and conflict because the information is scattered across many documents. The authors created a method that first guesses dataset mentions using a simple model, then uses a large language AI to check and fix these guesses. This corrected information trains the model better to find dataset mentions across many documents. Their system was tested on a set of over 1,700 passages and showed it can accurately find data mentions most of the time. This helps track how data is used and identify where data might be missing in these important reports.
What this means in practice
- •For humanitarian data teams: Automatically identify where datasets are mentioned in reports about displacement and conflict to improve data management and usage tracking.
- •For government policy analysts: Pinpoint dataset references in policy and operational documents to better understand data availability and gaps in fragile contexts.
Authors
Rafael Macalaba, Aivin V. Solatorio, Patrick Michael Brock, Olivier Dupriez
Abstract
Development and humanitarian organizations produce and support surveys, administrative registries, and other data resources to inform research, policy, and operations, yet systematically identifying where these datasets are referenced remains difficult. Such references are dispersed across research papers, project documents, humanitarian reports, and other unstructured text, limiting both the ability to trace data use and to identify potential gaps in data availability or dissemination. We present a weakly supervised framework for adapting dataset extraction to forced displacement and Fragile, Conflict, and Violence (FCV) documents without first constructing a large manually labeled training corpus. A lightweight model trained on general research literature generates candidate dataset mentions from unlabeled domain documents, which a frontier large language model (LLM) reviews in context, validating or rejecting candidates and correcting their extraction boundaries. The resulting annotations are supplemented with targeted synthetic and contrastive examples and used to fine-tune the lightweight model for large-scale extraction. We evaluate the resulting model on an independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents. Across the full benchmark, the model achieves 74.1\% precision and 70.5\% recall at the mention level; among passages containing dataset references, precision reaches 89.5\%. At the passage level, the model achieves 88.2\% accuracy and 88.6\% specificity in distinguishing passages with dataset references from those without them. These results demonstrate a practical approach for constructing domain-specific supervision when labeled data are limited, and provide a technical foundation for larger-scale analysis of data use and potential gaps in the displacement data landscape.