Temporal reasoning in large language models relies more on memory than calculation
Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs
Computation and LanguageArtificial Intelligence
Summary
People often think that understanding time in AI means doing math with dates or grasping cause-and-effect relationships. This paper shows that big language models don’t actually learn these skills separately. Instead, when tested, their improvement mainly comes from remembering common patterns about time, not from logical or arithmetic reasoning about it. So, current tests don’t really measure true time understanding but how well models recall typical time-related facts.
What this means in practice
- •For language model developers: Design evaluation methods that better distinguish memory recall from true temporal reasoning in language models.
- •For ai system testers: Avoid relying solely on existing temporal reasoning benchmarks when assessing models’ time understanding abilities.
Authors
Auss Abbood, Fernando Martínez-Plumed, José Hernández-Orallo, Nigel Collier
Abstract
Temporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and causal relations). This dichotomy shapes how benchmarks are designed, how models are trained, and how progress is measured, yet its empirical validity has never been directly tested. We provide two lines of evidence that challenge it. First, we test whether tasks labelled as arithmetic or semantic by the benchmarks support transferable gains within their purported mode. Finetuning on these subsets yields predominantly dataset-specific gains ($+6.8$ to $+23.1$ percentage points), while transfer to a different dataset in the same mode ranges from $-3.2$ to $+1.1$ percentage points. Across the two benchmark families used in the transfer analysis, this pattern provides little evidence of coherent, transferable temporal abilities. Second, using cognitive-demand-based model evaluation, we examine which capabilities actually drive performance on these benchmarks. We find that quantitative and logical reasoning, the assumed proxies for temporal arithmetic and temporal semantics, are not the primary drivers; knowledge is. Together, these findings show that current temporal reasoning benchmarks do not cleanly realise the arithmetic-semantics distinction and instead largely reward models for remembering how time typically works.