Table retrieval improves text to sql question answering across benchmarks
FERN: Cross-Benchmark Table Retrieval for Text-to-SQL with Frozen Retrievers and Learned Ranking
Information RetrievalComputation and Language
Summary
Text-to-SQL systems help turn plain questions into database queries but first need to find the right tables among many. The authors present FERN, a method that combines several frozen retrievers and ranks their results using learned features to pick the best tables. Compared to a prior method using large language models, FERN matches or approaches its recall rates on many question sets without extra fitting. Overall, it improves how well the needed tables appear near the top, with good speed and without costly model calls.
What this means in practice
- •For database application developers: Select relevant tables quickly for natural language queries using FERN’s combined retrieval and ranking approach to boost accuracy without retraining on new datasets.
- •For enterprise data integration teams: Improve data catalog searches across multiple pooled databases by implementing FERN’s feature-based ranking to enhance retrieval effectiveness while avoiding expensive large language model calls.
Authors
Arjun Rao, Nishant Gopinath
Abstract
Text-to-SQL systems over large pooled catalogs must first retrieve the tables a question needs. We compare against the published table recall of RASL, which uses Claude 3.5 models to extract keywords and select tables, on pooled BIRD, Spider, and Fiben catalogs. FERN (Feature-Engineered Ranking with No generative LLM) instead fuses three frozen retrievers into 100 candidates, describes each with up to 120 features, and reorders them with LightGBM rankers fitted once on BIRD's training questions. On Spider, where it was not fitted, its candidate pool alone reaches RASL's published recall at fifteen tables (98.0), and the ranker raises the share of questions with every gold table in the top five (complete@5) from 83.9% to 95.6%. At five tables, FERN's recall is within one point of RASL's on Spider (96.7 vs 97.0) and on BIRD with its evidence hints (96.6 vs 97.5), and 8.2 points below on BIRD without them (89.3). On 121 real BEAVER questions, without BEAVER fitting, its recall at fifteen (82.1) falls between two BEAVER-trained retrievers (80.8, 84.5), but its complete@15 is below both (49.6 vs 54.5 and 62.0). On Fiben, it falls far short at fifteen tables (54.2 vs 69.2, different question sets), although its recall at fifty is numerically close to RASL's question-only retrieval result (80.7 vs 79.4). On BIRD dev with evidence, non-retrieval features raise complete@5 by 7.7 points over retrieval signals alone. On BIRD dev, median latency is 208 ms per question on one A100, without hosted model calls.