Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot
2026-08-17 • Computational Engineering, Finance, and Science
Computational Engineering, Finance, and ScienceArtificial Intelligence
AI summaryⓘ
The authors propose a new, objective way to test systems that generate scientific research questions by using historical backtesting. They freeze a set of questions generated from old research and then check if future research answers or supports those questions without exposing the system to that future data. Their tests in astronomy show that methods focusing on evidence structure outperform those just relying on large language models (LLMs) alone. They also found that human and model judges disagree a lot, suggesting current evaluation methods are unreliable. The authors release their data and a future test setup to allow unbiased, time-based validation.
scientific research questionshistorical backtestinglarge language models (LLMs)evidence-structure-first generationtemporal corpus isolationevaluation metricsinter-rater agreementkappa statisticreproducible benchmarksastronomy datasets
Authors
Hui Mao
Abstract
Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future corpus then determines whether each question was subsequently answered, partially addressed, independently posed, or ignored, and whether its underlying premise was supported or refuted. The protocol is model-agnostic: any system that emits frozen questions can be scored. We release reproducible astronomy instances with temporally isolated corpora, frozen questions, auditable labels, four reference baselines, and a submission interface. Two findings result. First, evidence-structure-first generation outperforms LLM-only prompting: across a generator decomposition crossed with a four-cutoff stress test (2010-2024, 798 judged questions) whose last window postdates model training, LLM-only generation shows memorized relevance without specific foresight, while a generator using no model weights at all finds questions whose premises the future refutes in every era. Second, a seven-rater agreement study (two blinded human annotators, five judge models, 90 items) indicts the outcome taxonomy rather than the judge: two careful humans agree at kappa = 0.17, every judge model agrees with the professional annotator as well or better (0.17-0.26), and frontier models agree with one another at 0.60 -- certifying an LLM judge by model-model agreement would have overstated its reliability threefold. A prospective instance -- 200 questions frozen 2026-08-17, scored 2027-2030 -- is released so the central claims become contamination-free tests that time itself will grade.