Data readme improves finding related tables for large data lakes
Data.Readme: Documenting Table Relationships for Diverse Data Tasks
Databases
Summary
Finding the right tables in a huge collection can be hard because many tables look alike and lack helpful descriptions. The authors created Data.Readme, which figures out how tables are related in advance and saves this information. This helps computer programs quickly find and use the right tables for data tasks, making their work more accurate and faster. They tested it on big data collections and showed it works much better than other methods.
What this means in practice
- •For data engineering teams: Speed up and improve accuracy of selecting relevant tables from large data lakes for complex data workflows.
- •For business intelligence analysts: Access better guided table relationships to enhance multi-table queries and analysis with less manual exploration.
Authors
Jiaming Liang, Yeye He, Haoyu Dong, Zachary Ives, Surajit Chaudhuri
Abstract
Large language models (LLMs) can perform diverse data tasks when given a small set of relevant tables. However, for complex tasks requiring multiple tables over large data lakes, identifying relevant tables for the task becomes a prerequisite, but remains challenging. This is because there are often many tables in a data lake with similar table and schema names, and limited additional semantic context, thereby forcing agents to spend substantial time exploring tables at online query time, greatly affecting models' ability to perform tasks accurately, while increasing cost/latency substantially. We present Data.Readme, a novel table documentation approach that uses lightweight offline processing to infer and index rich \emph{table-to-table relationships}, including confusable, joinable, and union-able tables. We further develop a suite of agentic tools for LLMs to work with the index, to help agents retrieve relevant tables and perform tasks much more efficiently and effectively. Finally, we study how to optimize our index construction with a budget constraint. On data lakes containing up to 12K tables, Data.Readme improves task accuracy by as much as 40 percentage points across a range of data discovery, data analysis, and data science tasks, outperforming diverse baselines, including powerful coding agents with unlimited time and monetary budgets. We also show that equipping existing methods (such as coding agents) with Data.Readme can not only lead to consistent quality gains, but also significantly reduce their latency by up to 6x, demonstrating the potential of Data.Readme.