Large language models improve tabular predictions with text features
LLMs as Feature Engineers for Text-and-Tabular Prediction
Machine Learning
Summary
Predicting outcomes using tables with text and numbers is hard because the text is unstructured. The authors created a step-by-step method where one large language model (LLM) suggests useful text features, another extracts those features, and a prediction model tests their usefulness. They give feedback in regular language to guide the LLMs to better features. This approach speeds up finding good features and makes the predictions easier to understand by showing exactly which features mattered.
What this means in practice
- •For data science teams: Create interpretable categorical text features that improve prediction accuracy in combined text and tabular data.
- •For customer analytics teams: Generate transparent, semantically meaningful features from customer feedback text to enhance predictive models.
Authors
Merwan Barlier, Blaz Skrlj
Abstract
We introduce an iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models. To navigate the feature space, a generator LLM proposes semantic definitions, a separate extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance. We optimize this search by translating explicit model errors, such as AUC ranking inversions, into natural-language feedback, steering the LLM to resolve specific predictive failures. Evaluated across three public datasets, this error-driven loop accelerates feature discovery by up to $3\times$ compared to unguided search. Empirically, the generated features demonstrate strong multi-view complementarity, strictly outperforming any subset when combined with TF-IDF and dense embeddings. Finally, the framework guarantees instance-level interpretability: the discovered features dominate SHAP importance rankings and provide a fully transparent, semantic audit trail for every prediction.