Scalable tools find complex patterns and relationships in data tables
Scalable extraction and visualization of multi-attribute logical and functional dependencies in tabular data
Machine Learning
Summary
Finding how different pieces of information in tables relate helps in making better models and recognizing patterns. The paper introduces two tools that can discover not just simple exact rules but also more complex, conditional relationships between multiple columns. These tools work faster and can handle large tables with many columns. They also provide clear visual maps of these relationships. This helps people better explore data and check if artificial data mimics real data well.
What this means in practice
- •For data analysts: Discover complex attribute relationships and visualize dependency structures in large datasets for better data understanding and cleaning.
- •For data engineers: Enable scalable detection of conditional dependencies across many features to improve synthetic data validation and feature selection.
Authors
Chaithra Umesh, Arvind Lomrore, Neethu D, Kristian Seegel-Schultz, Saptarshi Bej, Olaf Wolkenhauer
Abstract
Understanding the structural relationships among attributes in tabular data is fundamental to machine learning and pattern recognition. While functional dependency (FD) discovery has been extensively studied, scalable discovery of logical dependencies (LDs), particularly as the number of attributes and dependency order increase, remains underexplored. These dependencies capture non-deterministic, condition-specific relationships among pairwise or multiple attributes. Furthermore, existing approaches do not provide a unified framework for extracting multi-attribute LDs and FDs. To address these limitations, we propose LDTool and HLDTool for extracting and visualizing multi-attribute LDs and FDs from tabular data. LDTool extends dependency discovery beyond pairwise relationships, while HLDTool enables scalable extraction through hypergraph-guided search-space reduction. Experiments on three simulated and eleven real-world datasets demonstrate that the proposed framework extracts meaningful LDs and FDs while improving scalability. LDTool recovers the same FDs as existing FD discovery methods with lower runtime in high-dimensional feature spaces, whereas HLDTool enables dependency discovery in datasets with hundreds of features. The proposed framework provides interpretable visualizations of dependency structures and supports applications in exploratory data analysis and the quantitative evaluation of synthetic tabular data.