Papers for

search system builders

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Evaluating historical search systems requires testing memory and data versions

A Historical Corpus Is Not a Historical System: Auditing Hindsight Leakage in Stateful Data Discovery

Abstract: Offline replay should estimate what a discovery system could retrieve at a historical point, yet freezing the corpus leaves interaction memory unconstrained. We formalize point-in-time (PIT) discovery through historical state $(D_t, θ_t, M_{< i})$ and introduce a paired replay that changes only memory availability. The protocol constructs PIT and full-stream Future views from behavior-only traces and audits selected entries with a Temporal Violation Rate. Across three table-text domains, two stream regimes, two retrievers, and five seeds (216,000 rows), Future inflated Asset Recall@100 by 2.62-5.24 points; all 12 paired intervals excluded zero. With behavior-only trace memory, PIT underperformed the no-memory Stateless condition; Future masked 32.7-48.4% of that harm. For a simulated positive-feedback cache, PIT added 4.65-18.96 points over Stateless while Future added another 4.11-9.58 points. On five timestamped FreshStack topics, Future exceeded PIT by 2.72 points [1.75, 3.71]. Historical evaluation must version and validate memory with the corpus.

Fri 11 SeptInformation Retrieval

The gist

When looking back at what a search system could find in the past, it’s not enough to freeze the data—it’s also important to consider what memory or history the system had available. The authors show that ignoring this memory can lead to overly optimistic results, hiding problems in how past searches would have actually performed. They introduce a method to test search accuracy by comparing setups with different memory access, revealing how memory affects search results in realistic scenarios.

Open → 2609.12766v1