Language models tested for checking timing of clinical events in health data
Testimony from Timelines: Evidence-Grounded Significance Testing of Temporal Clinical Hypotheses
Computation and LanguageInformation Retrieval
Summary
Many claims online talk about how things happen over time, like side effects following vaccines, but these claims often lack proper evidence testing. The authors studied how to make sure a computer program can check these claims correctly by carefully choosing which health records to consider and how to test the timing between events. They made a benchmark called STRATA with thousands of examples and a method called ART that helps a model explain each step it takes to verify a claim. Their tests showed that picking the right records is the hardest part, and ART helps close this gap, making the models more trustworthy for health data analysis.
What this means in practice
- •For hospital data teams: Automatically check and verify timing-related claims about vaccine side effects using clinical records to support safer health decisions.
- •For financial analysts: Test time-based sales hypotheses after product recalls by using evidence-grounded significance testing over transaction timelines.
Authors
Chengyang He, Wenlong Zhang, Koduvayur Subbalakshmi, Ping Wang
Abstract
Ask a language model whether sales tend to dip after a product recall, or whether one side effect tends to follow another after a vaccine, and it answers in a fluent sentence that reads like a finding, with no test behind it. Assertions about how events unfold across many cases fill the Web, and each is a temporal hypothesis that only a significance test over the right records can settle. The verifier must specify the null, select the test and decide which records bear on the hypothesis, and any of these choices can go wrong while the conclusion still comes out right. Yet benchmarks for language-model data analysis grade only the conclusion, with the relevant data handed over. We study the problem in health, where a wrong verdict is costly, and present STRATA, a benchmark that grades every decision on the way to a verdict: 8,735 hypotheses over the clinical timelines of vaccine adverse-event reports, with gold plans, evidence sets, p-values and verdicts derived under a released codebook and reproduced by a deterministic oracle. We also propose ART (articulate, retrieve, test), a staged paradigm that exposes each decision as an inspectable output. Extensive experiments with five model configurations show that systems fail mostly at choosing which records count, a verdict-evidence gap that accuracy alone conceals. ART closes most of this gap and brings a 31B open model to within 0.01 of the frontier models under the same paradigm; the frontier models succeed once given the codebook.