Summary
Many language models can write code to analyze cause-and-effect questions from published economics studies, but this code doesn't always produce the right answers. The authors created a test called CausalVerify that checks whether the code actually finds the correct causal effect, not just whether it runs. They tested several language models and found wide variation in success, with some models often getting the wrong answer even when the code runs without errors. The researchers also found that just looking at the code's instructions or how confident the model is doesn't reliably show if the result is correct. This benchmark focuses on specific types of causal analysis and standardized workflows.
causal inferencelanguage modelsdifference-in-differencesinstrumental variablesregression discontinuityevent studyeconometricsbenchmarkcode executiontreatment effect
Authors
Yonghong Zhang, Ricardo Correia, Isabel M. Parra, Yong Xie
Abstract
Existing causal-inference benchmarks for LLMs mostly score method descriptions or whether generated code runs, not whether the executed workflow recovers the target causal estimate. CausalVerify studies this verification problem for structured econometric causal-estimation workflows by separating realistic interpretation from verifiable computation. It pairs 259 published economics papers (reconstructed research question, data description, institutional context) with 100 fixed-seed synthetic scenarios that realise CSV datasets for difference-in-differences, event study, instrumental variables, and regression discontinuity designs. Experiment A (real-paper text agreement) scores method-family and direction agreement against four-LLM consensus labels. Experiment B (synthetic execution) runs model-written R code and checks whether the extracted treatment-effect estimate matches a canonical estimator on the same realised dataset; this execution-grounded correctness layer is L2b+, distinct from L2b, which records only whether the code executes. A calibration arm asks whether self-reported confidence separates correct from incorrect workflows. On Experiment B, seven LLMs reach L2b+ pass rates of 10% to 88% at the default 50% tolerance, and 66 of the 426 workflows that execute (15.5%) return a wrong estimate. Execution ranking (L2b) agrees with L2b+ far better than text-direction scoring (L4): Kendall $τ=0.81$ and Spearman $ρ=0.93$, versus Kendall $τ$ between $-0.20$ and $0.10$ for L4. Llama-3.3-70B-Instruct shows the same qualitative gap, and reported confidence does not reliably separate correct from incorrect workflows. The claims are confined to standardized single-shot workflows in these four design families under the evaluated R backend and model panel; the benchmark does not measure general causal-inference ability. Code, data, cached outputs, and a datasheet are released.