Current methods fall short on extracting insights from large table collections
Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?
Information RetrievalArtificial IntelligenceComputation and LanguageDatabases
Summary
Many people want to get useful information from huge collections of tables but existing tools only solve parts of the problem. The authors looked at different research areas and combined their ideas into one clear framework called Open Tabular Insight Extraction. They found that current systems mainly focus on analyzing tables but don’t cover all the steps needed from start to finish. They suggest new directions for building better tools and tests that really check if these tools help users find insights.
What this means in practice
- •For data engineering teams: Design new workflows to integrate multiple methods for comprehensive insight extraction from large collections of tables.
- •For business intelligence developers: Create evaluation tools that test insight systems across diverse datasets without relying on prior knowledge of the tables.
A position paper. It proposes an approach and reports no results.
Authors
Daniel Gomm, Maarten de Rijke, Madelon Hulsebos
Abstract
Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person's insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at https://open-tabular-insight-extraction.github.io.