Data composition system organizes diverse data lakes for better analysis
DIADA: Automatic Data Composition in Data Lakes
Databases
Summary
Data lakes store lots of different pieces of information scattered across many tables, making it hard to find which pieces belong together. The authors introduce DIADA, a system that automatically groups related data by checking how different attributes depend on each other, without focusing on a single task. This helps create meaningful data groupings that can be used for many kinds of analysis, improving confidence in results without extra manual effort. Their method can find complex relationships that older techniques might miss, making data lakes easier to use for many purposes.
What this means in practice
- •For data engineering teams: Create reusable, task-agnostic data groupings from scattered raw data to support multiple analytic workflows without manual attribute matching.
- •For business intelligence teams: Improve confidence in analytics and modeling by using automatically composed attribute groups that reduce noise and highlight meaningful relationships.
Authors
Marc Maynou, Albert Martin, Sergi Nadal, Anna Queralt, Oscar Romero
Abstract
Data lakes contain a plethora of attributes scattered across many tables that, when combined, provide enhanced assets for data analysis. Nonetheless, deciding which attributes belong together in meaningful relations remains a manual, per-task effort. Merging by joinability alone provides no guarantees regarding attribute relevance, while selecting features against a single target discards attributes useful to other tasks. To address this gap, we introduce the data composition problem: organizing a fragmented, heterogeneous lake into meaningful relations, agnostic of any particular analytical task so that the resulting organization can serve as a common foundation for diverse downstream analyses. We propose DIADA, a composition system that employs multivariate dependence as the criterion for assessing the meaningfulness of a relation and approximates it by hypothesizing independence among attributes and identifying those sets that violate this hypothesis. To do so, we map the attributes to a predicate space, forming a lattice under inclusion and mining those predicate sets that exhibit dependence among their constituents. We contribute a dedicated and scalable algorithm to effectively explore this space, outscaling classical algorithms for mining relationships, thus discovering dependencies that would otherwise be impractical to identify. We demonstrate that applying a single data composition process benefits diverse potential downstream tasks. This is the result of providing a subset of low-noise, statistically relevant attributes that increases the confidence that detected patterns are grounded in real relationships, thus preventing common modeling issues in large-scale environments.