Decision tree learning improved by large language models for tabular data

From Prompts to Trees: Effective LLM-Guided Tree Generation for Few-Shot Tabular Classification

Machine LearningArtificial IntelligenceComputation and Language

Summary

Classifying data stored in tables can be tricky when there isn’t much data to learn from. Large language models (LLMs) know a lot but are slow and hard to understand when used directly for this task. The authors found a way to help computers learn better by asking LLMs to first make simple rules and then organize those into decision trees, which are easier to follow and faster to run. This new method works better and uses less effort than previous approaches, making the results both more accurate and transparent.

What this means in practice

  • For data scientists: Create more interpretable and accurate decision tree classifiers when only limited tabular data is available.
  • For business intelligence teams: Build faster and more understandable decision tree models for customer or sales data with minimal human labeling.

Authors

Yue Qiu, Zekang Du, Yiqun Diao, Bingsheng He, Qinbin Li

Abstract

While Large Language Models (LLMs) possess rich world knowledge and impressive generalization capabilities, their direct application to tabular data classification is hindered by high inference costs and limited interpretability. In contrast, decision trees are fast and transparent but often underperform in low-data regimes. In this work, we propose a novel framework that bridges these paradigms by distilling LLM knowledge into interpretable decision trees under a few-shot learning setting. Instead of directly prompting the LLM to generate full trees, which is often unstable and inefficient, we develop a three-stage paradigm that prompts the LLM to generate rules and organize the rules into a tree. Experiments on multiple real-world tabular datasets demonstrate that our method achieves superior accuracy and interpretability with significantly lower prompting overhead compared to existing baselines.
View PDFOpen arXiv