Hidden dates in prompts cause large changes in ai model scores
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Computation and Language
Summary
The paper shows that many AI language models secretly get today's date added into their instructions, which affects how well they perform on tests. This hidden date changes daily, causing the models to give different answers on different days, even though everything else is the same. These changes can affect test scores by a lot and even reorder which models seem best. The authors found that usual ways of testing don’t fix this problem, meaning it’s a hidden factor that makes comparisons between models less reliable.
What this means in practice
- •For ai development teams: Design evaluation pipelines that control or account for hidden date inputs to improve consistency in model testing.
- •For machine learning engineers: Adjust deployment and testing procedures to avoid daily date-driven score fluctuations for more stable product behavior.
Authors
Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense
Abstract
Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection of the current date into system prompts, which users cannot control and which changes every day. Across 9 recent LLMs and 6 datasets spanning multiple-choice QA (MCQA), math reasoning, code generation, and machine translation, performance varies solely with the current date, with deltas of up to 6% on MCQA, 14% on math reasoning, 7% on code generation, and 2.84 BLEU on machine translation. Model rankings also shift, affecting leaderboards. This date effect exceeds other sources of non-determinism, such as batch size and numerical precision. Standard prompting techniques -- chain-of-thought and few-shot prompting -- do not reduce the sensitivity; chain-of-thought even amplifies it. Our findings underscore the need for careful evaluation protocols to ensure reproducibility and fair comparisons in LLM research.