Summary
Accessing useful insights from big collections of tables, like data lakes, is a difficult but important task. The authors show that current research on this problem is divided into separate groups that don’t always connect well, each using different terms and focusing on specific parts. They create a new framework called Open Tabular Insight Extraction (OpenTI) to unite these groups and explain how to get the information people really want from many tables. They also look at current tools and tests and find that most only cover part of the process and are not designed for open, real-world situations. Finally, they suggest future directions to build better systems and ways for people to interact with table-based information.
table question answeringtext-to-SQLdata analysis agentsinsight extractiondata lakesinformation retrievalnatural language processingmachine learningbenchmark evaluationhuman-computer interaction
Abstract
Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person's insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at https://open-tabular-insight-extraction.github.io.