Papers for

enterprise analytics teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Experience guides adaptive search to cut data lake workload

Learning the Lake: Reliable Experience for Adaptive Data Product Discovery

Abstract: Data-product discovery searches a full lake even when workloads revisit related products and regions. Repetition permits contracted search, but similarity cannot justify a route because one omitted asset invalidates a conjunctive product. We study when serving experience can safely reduce this work. Evolving Discovery Memory records source-labelled query--product--region evidence above a fixed regional index. SafeLake separates operational familiarity, which determines how much to search, from independently calibrated product evidence, which determines where to search. The fixed-probe comparison holds the adaptive budget constant between SafeLake and Familiarity-only. On TAT-QA, product steering raises Product Recall by 0.072; ConvFinQA shows no resolved map gain, while the HybridQA sensitivity favors Familiarity-only in Full R@100. Trace-only, missing, and false feedback expose boundaries on map steering, while scope-audit agreement cannot certify the source. Across clean confirmed-feedback streams under the frozen transductive protocol, the formal controller saves 49.5--82.7% of cumulative asset exposure. Experience determines when to contract; reliable evidence determines where to contract.

Fri 11 SeptInformation Retrieval
The gist
Searching through data lakes to find related products often wastes time because it scans too much, even when many searches have similar targets. The authors show how learning from past search experience can safely reduce the amount of data scanned without missing important results. Their method, SafeLake, smartly decides how broadly to search and where to focus effort by balancing familiarity with verified product evidence. Tests show it can save nearly half or more of the workload while keeping recall high.
Open 2609.12754v1