Papers for

econometrics data teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Causalverify benchmarks llms on executed causal estimation workflows

CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows

Abstract: Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $τ=0.81$ and Spearman $ρ=0.93$, versus Kendall $τ$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.

Mon 7 SeptArtificial IntelligenceComputation and Language
The gist
It is hard to tell if AI models that write code for causal studies actually produce correct results when that code runs on real data. The authors created CausalVerify, a test set that uses fake data based on real economics studies to check if AI-generated code recovers the right causal effect, not just if it runs. They found large differences in model performance and that some common checks do not reliably identify correct answers. This helps understand where AI tools for causal analysis work well or fall short.
Open → 2609.07944v1