Enriching datasets with detailed variable metadata to check model fit
Reading the Data Back: Enriching Variable-Level Metadata for Model-Data Consistency Checks
Digital LibrariesDatabases
Summary
Data repositories often don’t include detailed information about each variable, which makes it hard to check if a statistical model fits the data properly. The authors created a way to add this missing information as metadata linked to the models, analyses, and variables used. They tested methods to extract this metadata automatically from statistical code and papers, then used it to flag models that might not match their data well. Their approach helps make data and models easier to review and understand together.
What this means in practice
- •For data archive managers: Add detailed, linked variable metadata to datasets to enable automated checks of model-data consistency during data publication.
- •For statistical software developers: Include metadata extraction features in tools that connect datasets to analysis code for improved reproducibility and review.
Authors
Eryk Kulikowski
Abstract
Purpose: Research data repositories often lack variable-level metadata. Model-choice screening, checking whether a reported model suits the values of its outcome variable, also requires summaries of those values and links to the analyses that use them. We ask how repositories can represent and acquire them as metadata with evidence and review histories. Methods: We propose a metadata application profile compatible with the Data Documentation Initiative Cross-Domain Integration (DDI-CDI) model, linking versioned variables and empirical profiles to reported analyses, estimators, and analysis roles. Extraction provenance and review decisions are recorded separately. We evaluate streaming data profiling, deterministic extraction from Stata and R code, and language-model extraction of analysis records from papers. Results: A worked export of one replication package illustrates the profile: it passes shape and interchange checks and answers four queries, one of which returns the analyses that raise no alert. Screening 4,868 replication datasets from six political-science journals links outcomes to profiled columns in 1,440 deposits and flags 965, including 14 of the 21 deposits with a hand-verified linear model on a count, proportion, binary, or ordinal outcome. AI-assisted adjudication yields 55% precision on a near-balanced sample of 119 count and proportion candidates; screening rules were developed on the same corpus. Conclusion: The profile makes analysis-variable relationships queryable while preserving evidence and review history; screening yields review candidates with context. Code, metadata, and measurements are released.