Experience guides adaptive search to cut data lake workload
Learning the Lake: Reliable Experience for Adaptive Data Product Discovery
Information Retrieval
Summary
Searching through data lakes to find related products often wastes time because it scans too much, even when many searches have similar targets. The authors show how learning from past search experience can safely reduce the amount of data scanned without missing important results. Their method, SafeLake, smartly decides how broadly to search and where to focus effort by balancing familiarity with verified product evidence. Tests show it can save nearly half or more of the workload while keeping recall high.
What this means in practice
- •For data engineering teams: Reduce resource use during complex data lake product searches by adapting search scope using past query experience and verified evidence.
- •For enterprise analytics teams: Improve efficiency and accuracy of data product retrieval in analytics workflows by leveraging adaptive search guided by operational familiarity and product evidence.
Authors
Yixi Zhou, Fan Zhang, Sikun Wang, Yingfan Xu, Haipeng Zhang
Abstract
Data-product discovery searches a full lake even when workloads revisit related products and regions. Repetition permits contracted search, but similarity cannot justify a route because one omitted asset invalidates a conjunctive product. We study when serving experience can safely reduce this work. Evolving Discovery Memory records source-labelled query--product--region evidence above a fixed regional index. SafeLake separates operational familiarity, which determines how much to search, from independently calibrated product evidence, which determines where to search. The fixed-probe comparison holds the adaptive budget constant between SafeLake and Familiarity-only. On TAT-QA, product steering raises Product Recall by 0.072; ConvFinQA shows no resolved map gain, while the HybridQA sensitivity favors Familiarity-only in Full R@100. Trace-only, missing, and false feedback expose boundaries on map steering, while scope-audit agreement cannot certify the source. Across clean confirmed-feedback streams under the frozen transductive protocol, the formal controller saves 49.5--82.7% of cumulative asset exposure. Experience determines when to contract; reliable evidence determines where to contract.