Large language models cannot reliably report their financial knowledge over time
LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Computation and Language
Summary
The paper shows that large language models like GPT-5 often do not know when they actually know information about financial events. Models may claim ignorance but still answer correctly, or say they know things even after the known knowledge cutoff dates. The authors created FinDendro, a test with thousands of dated financial facts, to check if models can accurately self-report what they know at different times. They find that self-reports and instructions about knowledge dates are not trustworthy for ensuring predictions are leak-free during backtests.
What this means in practice
- •For quantitative finance teams: Assess the validity of financial backtests by selecting model evaluation periods based on empirical knowledge boundaries rather than stated cutoffs.
- •For ai model evaluators: Use FinDendro to measure and monitor temporal knowledge limits of large language models to improve transparency about what the model truly knows and when.
Authors
Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao, Jiarui Chen, Zehan Li, Jimin Huang, Alejandro Lopez-Lira, Yuehua Tang, Sophia Ananiadou, Jian-Yun Nie
Abstract
Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.