Dataset search improves by matching user intent with table content
Bridging Search Intent and Tabular Data in Dataset Search
Information RetrievalDatabases
Summary
Finding the right dataset to answer a question is hard because existing search tools mostly look for keywords and miss what users actually mean. The authors created a way to understand user goals better by imagining what the ideal dataset table should look like and then searching based on that idea. They also built a tool to filter datasets using natural language descriptions of dataset features. Tests show these methods find more relevant tables than existing search techniques, making it easier to find the data you need.
What this means in practice
- •For data analysts: Search for datasets that match specific analysis tasks by using natural language queries and intent-aware retrieval.
- •For business intelligence teams: Filter large dataset collections with flexible natural language constraints to quickly narrow down relevant tables for reports.
Authors
Wenjing Lin, Madelon Hulsebos
Abstract
Dataset search remains a bottleneck in data analysis workflows, as existing systems largely rely on keyword matching or shallow retrieval that fail to capture user intent. To facilitate intent-driven search, we present two retrieval components to equip multi-faceted dataset search interfaces through natural language. First, we propose Hypothetical Schema Embeddings (HySE) that uses a language model to generate task-aligned hypothetical table schemas which are used for embedding-based retrieval, bridging the gap between intent and table content. We introduce KaggleDS, a corpus comprising 6,503 tables and diverse task-queries derived from Kaggle, and show that HySE consistently improves recall over strong keyword and embedding-based baselines, as well as over a dense table retriever fine-tuned on KaggleDS. We also show that HySE generalizes to end-to-end question answering on, for example, the NQ-Tables dataset, a corpus of 169,898 Wikipedia tables with human-written queries, where HySE improves Recall@10 by +6.0 pp over dense retrieval baselines. Second, we introduce FlexPrune, which translates dataset constraints expressed in natural-language into SQL predicates over a customizable metadata index, enabling filtering across arbitrary metadata attributes. Combined with HySE, FlexPrune increases Recall@10 by +6.0 pp over dense retrieval followed by tag-based filtering. Our results show that natural-language-based retrieval supporting task and metadata queries offers a viable alternative to schema-level keyword search in dataset search.