Evaluating historical search systems requires testing memory and data versions
A Historical Corpus Is Not a Historical System: Auditing Hindsight Leakage in Stateful Data Discovery
Information Retrieval
Summary
When looking back at what a search system could find in the past, it’s not enough to freeze the data—it’s also important to consider what memory or history the system had available. The authors show that ignoring this memory can lead to overly optimistic results, hiding problems in how past searches would have actually performed. They introduce a method to test search accuracy by comparing setups with different memory access, revealing how memory affects search results in realistic scenarios.
What this means in practice
- •For data engineers: Ensure accurate historical search testing by versioning both data and system memory state when validating discovery tools.
- •For search system builders: Improve search performance evaluation by auditing how memory use inflates results compared to memory-less baselines in data streams.
Authors
Yixi Zhou, Fan Zhang, Sikun Wang, Yingfan Xu, Haipeng Zhang
Abstract
Offline replay should estimate what a discovery system could retrieve at a historical point, yet freezing the corpus leaves interaction memory unconstrained. We formalize point-in-time (PIT) discovery through historical state $(D_t, θ_t, M_{< i})$ and introduce a paired replay that changes only memory availability. The protocol constructs PIT and full-stream Future views from behavior-only traces and audits selected entries with a Temporal Violation Rate. Across three table-text domains, two stream regimes, two retrievers, and five seeds (216,000 rows), Future inflated Asset Recall@100 by 2.62-5.24 points; all 12 paired intervals excluded zero. With behavior-only trace memory, PIT underperformed the no-memory Stateless condition; Future masked 32.7-48.4% of that harm. For a simulated positive-feedback cache, PIT added 4.65-18.96 points over Stateless while Future added another 4.11-9.58 points. On five timestamped FreshStack topics, Future exceeded PIT by 2.72 points [1.75, 3.71]. Historical evaluation must version and validate memory with the corpus.